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SECTION 1 



1,1 aackyround 

In 1982 a stix^ was perfonned for the U.S. DQ>artroent of Ehergy 
Teclr.ical Information Center (now Office of Scientific and Technical 
Infomfltion) to investigate the value of information geierated f rem DOE R&D 
funding and the contribution that the Ehergy Data Base and its derivative 
services and products nake to the value of this information, "flie results 
of the study danonstrated that energy information has substantial value 
irtien considered from the stanc^ints of asnount of use, purposes of use, and 
value measured by yttiat usszs are willing to pay for the information and the 
savings in lalx>r, equipaient, etc. derived from use of the information. 
Furthermore, energy information services and products derived from the 
Eiiergy Data Base clearly contribute greatly to the value of energy 
infomation fcy making it more accessible and, therefore, more usable. 

Ohere are two ways to look at the value of energy information. Olie 
first perspective involves the $5.8 billion exgended by DOE on research and 
developinent (in 1981). The return on this investment is achieved through 
the accai?>lislinent of ^secific goals and objectives of the R&D and through 
knowledge that is directly obtained from the research results. The know- 
ledge is I'lso g2dned by others through information ccninunication and use. 
It was estimated that proximately 60,000 energy scientists and engineers 
(f aided by DOE) read about 14 million joumad articles and r€|»rts in 1981. 
These readings were used for matjy purposes such as research, professional 
developnent, education of others, writing, and managanent. A survey of 
energy scientists and engineers showed that many of the readings led to 
savings of scientists' time !:i\d/or equipment used in their research. For 
readings of articles there was estimated to be an average savings of $590 
per reading and for technical r^rts there was an average savings of 
$1,280 per reading, ito extrapolated to the populati^ of energy scien- 
tists and mgliieers, the total savings yielded froc^ reading came to about 
$13 billion just for 1981. Ihus, energy information production, distribu- 
tion and use accomts for substantial future savings to EOE scientists. 
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Presunably, the more energy informaticn is used, tlie greater will be the 
return on investnient yielded from R&D eatpenditures. 

Bie second way to lode at the value of information is what would 
happen if the information was not available. Uie EOE Technical Information 
Services program managed by the Office of Scientific and Technical Informa- 
ticn {OSn) is designed to provide greater access to energy infonraticn, 
Uiis is acoooplished by making copies of energy technical reports available 
and by providing bibliographic services and products for managing, identi- 
fying and locating energy publications. These services and products were 
investigated in 1981 to determine their contribution to the value of energy 
Information. A very large portion of the 14 million readings mentioned 
above resulted from use of OSTI services and products. A calculation was 
.nade of the amount of readings that would be lost if these services and 
products were not available. This calculation was made by estimating the 
average cost per reading from substitute services and products, which 
nearly alviays was higher than the average cost of per reading from existing 
services and products. Thus, at a fixed total budget, fewer readl'igs could 
be achieved. Ihe loss of readings would, hypothetically, result in less use 
of the information and therefore, in less value derived from the 
information. 

It was estimated that the corresponding lost value, that is value 
attributable to the Ehergy E&ta Base services and prodix:i:s, was about $117 
million in term of the value determined by what users were willing to pay 
for the energy information (in price paid and time ejqsended in getting and 
using the information) . Furthermore, in terms of savings, it was estimated 
that the value was about $3 billion for the value measured by the savings 
in scientists' time and equipment resulted from reading energy publica- 
tions. Ti -» $3 billion savings in labor and equipnent frcni the EEB products 
and services can be roughly translated into productivity. If it is assumed 
that, with the EEB, research and develc^xiwnt costs $5.8 billion for a given 
level of output, without the ElB the same output would require an invest- 
ment of $8.8 billion. This is an increase in productivity of eibout 52 
percent. This says that to accoiplish the same R&D output without TIC 
information services, the R&D budget would have had to have been $3 billion 
higher. 

2 11 



The study results were widely disseminated and discussed by 
economists, the information research coramunity and government policy 
makers. The stud^ methods were slIso replicated with similar findings in 
the Department of Defense and a large corporate information system. 

It wsis felt that the methods and models esiployed in the previous 
studies ought to be refined and an additional investigation made to deter- 
mine what contribution intermediary information transfer organizations such 
as libraries and Information analysis caters make to the value of informa- 
tion. Puno, other factors such as timeliness of primary publishing and 
distribution and cocpr^&isiv&iess of the Ehergy Data Base were considered 
to be important as well. This further investigation was made under a grant 
from the Nationa? Sc.'ence Foundation with partial contribution by the 
Department of Eiiergy, Of /ice of Scientific and Technical Information. Hhis 
volume provides the result of hew much typical libraries that serv^e profes- 
sionals such as scientists and engineers contribute to the value of 
information. 

In conducting the studjy we concentrated csi library services pro- 
vided at three locatiais: Rocky Flats, Rockwell Ehergy Systems Group and 
Oak Ridge National r«iboratory (ORNL) , We also obtained inloriTBtion from a 
general survey of scieitists and engineers found in nine fields of science, 
'The basic approach in this investigation involves estimating the value of 
energy information using a slightly revised method of calculating value 
than that used in the previous lepott. 

Section 2 of this report, gives an estimate of the amount and pur- 
pc' of reading of journal articles, technical reports and books in the 
three locations. The time spent reading is also estimated, which is a 
major contributing factor to what users are willing to pay for inforrodtion 
read. Also, value of information is estimated in terms of what users are 
willing to pay for it. Finally, value is estimated in terms of v^t 
ngs (in time and equipment) are achieved from reading. 
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Hye anxxnt of V2di^ attributable to the libraries serving the three 
locations above is then estimated. Section 3 prosrides these estimates 
vhich are derived from the amomt of reading resulting from the collections 
and services provided by the three libraries: Rocky Flats Technical 
Library* Itockwell &»'}rgy S^ens Groip Library and OFML Library System. 
Alternative ^xirces of information are determined (or subjectively chosen) 
assuning that the libraries and their services are not available. HiCn 
costs are estimated for these substitutes of the library services. Ihese 
sibetitutes are nearly always more expensive (or have vorse performance) . 
If we eissime a given amount of expenditures will be made, the amount of use 
of information must decrease (because the cost per reading goes ip) and, 
therefore, vzCLue will also decrease (in terms of v^t users are willing to 
pay and savings derived f rcsn reading) . inlying the average values and 
savings derived to the nuniber of lost readings (due to more ea^sensive 
substitutes) provides the estimates of the contribution that library 
services make to the /alue of informaticsi. 

In Section 4, we give results which describe the performance and 
effectiveness of library services in the three locations. Performance is 
measured in terms of (1) the timeliness of services provided and materials 
delivered and (2) quality of services. Effectiveness is measured in terms 
of amount of use of specific services and products. Cost effectiveness is 
given in terms of the cost per use. Of course, value is dependent on 
amount of use of the materials and services. Finally, in Section 5 we 
atteBf)t to relate libraries capabilities to performance, performance to 
effectiveness and perfomance to value (all for search services) . In the 
case of the relationship between performance and value we use a method, 
conjoint measurement, to relate relevance of online search results and 
speed of response to veO-ue in terms of user time. 

There are two other volumes that report factors that value of 
information is dependent Lpon. Volune 2 gives results of a stu<^ of two 
software Infomaticn Analysis Centers 2md the value of software tiiat is 
attributable to the centers. In Volune 3, we investigate the inportance of 
timeliness to vzilue of infomation found in technical reports and journal 
articles. Timeliness is analyzed from the 3tan(^int of the delay of pub- 
lishing, delay in getting copies of technical reports and articles, delay 



in publishing bibliographic databases, and delay in conducting reference 
searches cn-reguest. 'Stat volume also looks at the value of non-U. S. 
materials and the cross-cuttii^ nature of the subject fields required by 
rO£ scientists and engineers. 



1.2 Stut^ Methods 



Data were collected fron three surveys: 



(1) A survey of the pc^xilation of scientists and engineers fron 
nine f iel<fe of science .was conducted as part of a study being 
performed for the National Science Foundation (Statistical 
Indicators of Scioitif ic and Technical Comiujnicaticn) . This 
survey involves estinating the extent of authorship of journal 
articles, jcHirnal reading, journal subscriptions, technical 
report reading, library use, numeric data base searching and 
bibliographic data base searching. Deniographic information 
identifies scieitists and engineers are funded by DOE. 

(2) A survey of the professionals ecplcyed at Rocky Plats, Rockwell 
and ORIffi was also conducted. ISiis general information usage 
survey obtained infonnaticHi about generzd reading, library use, 
awareness of services and satisfaction with services. It is 
noted that this survey included managers, adninistrators and 
(^rational professionals as well eis scientists and engineers 
engaged in R&D. 

(3) In addition, four surveys were conducted at the three library 
locations to determine detailed information about recent uses 
of ^)ecific library materials and services: ciiline searches, 
•manual reference searches, journal articles and technical 
r^rts. Data collected included sources of services, value of 
reading, purposes of use, other value related information and, 
for the two types of searches, conjoint measurement data. 



TSie survey of scientists and «>gineers actually was done in two 
parts. Ttie first part obtainL-d information about journal article author- 
ship, reading (i.e., amount, means of identification and access, purpose of 
reading and value derived frcm reading), and journal subscriptions, The 
second part ccaicentrated on technicatl report reading, library use, and 
online searching fron nuneric and bibliographic da t abases. Samples for 
both parts were chosen randociJ.y fron lists of scientists and engineers 
provided by professional socie.ies. The first part involved a mail-out of 
1,958 questionnaires, of v^ich ,252 responded (64 percent response rate). 
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Of the»^. 588 of the scientists and engineers indicated they were not 
involved, thus yielding 664 questionnaires that were anzdyzed. The second 
part had a mil-oiit of 1,017 questionnaire of i^ich 467 reqxxises were 
received and 195 were analyzed. Ilie second survey involved IXX funded 
professionals (scifeicists and engineers and others) ^^o work at Reeky 
Flats, Rockwell Qiergy Systems Groi^ and Oak Ridge Kitional Laboratories. 
The professionals were rsmdomly chosen from lists provided by the 
organizations mentioned above. A total of 200 questionnaires were 
distributed and 137 received (69 percent response rate) . To achieve this 
level of response the questionnaires were distributed by the libraries (or 
KRI) to a random saii^)le of professionals who work at the three locations. 
Sane data were collected by telephcaie follow-t?) in order to achieve desired 
re^xsnse rates. The carpleted questimnaires were returned to KRI by the 
respondents by a self-addressed envelope in order to ensure 
confidentiality. 

1.3 Sumrary of Results 

The average annual number of readings per professional by 5,188 
professionals at Rocky Flats, Rockwell and ORNL is: 

• 99 readings of journal articles 

• 44 readings of techniccil reports 

• 15 readings of books 

!nie average time ^>ent annually reading by these professionals and the 
corresponding cost of this time is: 

• 95 hours ($2,600) reading journal articles 

• 75 hours ($2,100) reading technical reports 

• 63 hours ($1,700) reading books 

• 233 hours ($6,400) total reading, 

Hie principal purposes for this reading are as follows (given in ranked 

« 

order): proiassional developnent ; applying findings to a current project; 
applying methods to a current project; writing; preparing proposals; 
preparing lectures or presentations; and planning, budgeting and 
management. 

6 15 



The three libraries are used extensively by energy professioials, with 
average annual uses enconpassing: 

• 25 uses of the libraries 

• 21 times visited one of the librari;^ 

• 17 tiroes read materials in the library 

• 10 tiroes photocopied materials in tlie library 

• 11 tiroes checked materials out 

• 8 times obtained assistance in locating materials frcm a 

library staff person 

• 4 times (Stained assistance in identifying relevant ne^^-iials 

from a library staff member 

• 4 times used microform or microfiche equipment 

• 2 times obtained materials from other libraries 

• Ckice did a search on library equipment 

• Less than once obtained cm OTiline bibliographic search and used 
audio visual equipmen';. 

T^B amount of annual readings of library matorials from all three 
locations is as Eollws: 

• Journal Articles 

- 53% of all readings came from library materials 

- 53 average annual readings per profess icml 

• Technical Repocts 

39% of all readings came ftm library materials 
17 average annual readings per professional 

• Boc^s 

39% of all readings came from library materials 

- 6 average annual readings per professional. 
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Bie reasons for extensive use of library materials and services are; 

• Hig^ a^reness of ^rvices provided 

• Distance f ran library averages less than 5 minutes 

• Library provides access to older materials: 

- 3.5 years average age of journal articles read from library 
oc^ies 

0.6 years average age of articles read from personal 
subscriptions 

1.5 years average age of articles read frcin other sources 

2.4 years average age of technical reports read from 
library cc^ies 

1.2 years average age of technical reports read from other 
sources 

• Some library journal cc^ies are frequently read by profession- 
als rather than subscribing to the journals because of the 
journal price an^oz because they are readily available frc3ir» 
the library. 

The average value of reading library materials in terms of what 
users are willing to pay is: 

• $41 for reading a journal article 

• $102 for reading a technical report. 

The total value at the three lcx;ations of v^t users are willing to pay for 
readings of library journal articles and technical reports cones to: 

• $16.6 million, or 

• $3,200 per profespic»)al, 

T*ie average value at the three locations of reading library materials in 
terms of savings to users is: 

• $385 for reading a journal article 

• $706 for reading a technical report. 



•flie total value at the three locations of reading library materials in 
tenQs of savings is: 

• $144 million, or 

• $27,700 per professional. 

Bie values attributable to library services are as follows: 

• Value in terms of villingness to pay 

$3.1 million total value attributed to libraries, or 
$600 per professional 

• \^lue in terms of savings derived from reading 

$28 million total value attributed to libraries, or 
$5,400 per professional. 

Hie performance of the libraries is considered to be exenplary in 
terms of timeliness and quality of service. 

Effectiveness meaisured in terms of user satisfaction is also 
considered very high. 

Indications are that performance of online searching is related to 
the reference librarian's subject knowledge and search skills. Anount of 
use is related to performance measured in terms of tiineliness and quality 
of searching. Value in terms of willingness to pay is clearly related to 
speed of response to search questions and relevance of search results. 
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SECTION 2 

OHE USE AND VALUE OP ENEHGY IKfOFMKTION 

2.1 The Dae of Bierqy Information 

Kiere are approximately 5,200 professionals who work at Rocky 
Plats, Rockwell Qiergy Systenis Group and Oak Ridge National Laboratories 
(OFNL) . Ihese professionals were surveyed by taking a systematic random 
sample (with randan start) from recent telephcxie listings, A total of 137 
perscxis responded to 200 questionnaires sent out. While it is estimated 
that most (51%) of these professionals are scientists and engineers engaged 
in research and develcpioit, many of the professionals £dso have as their 
primary work role such activities as management and adninistration (27%), 
operations (12%), education and training (2%), and other activities (8%). 
These professicxials are geierally well educated. Of the respondents to the 
general information use survey, ^en cisked about highest degree earned, it 
was found that 14 percent had Doctorate degrees, 25 percent had Master's 
degrees, 44 percent had Bachelor's degrees and 17 percent had Associate 
degrees or less. 

The professional staff of Rocky Flats, Rockwell and ORNL clearly 
read a great deal even though not all professionals are scientists and 
engineers Bigaged in P^D. Table 1 gives the average and total nunber of 
readings of journal articles, technical reports and books by these 
professionals. The average annual number of readings of journal articles 
by the professional staff at all three locations is 99 per person. This is 
scinewhat less than the average of 124 annual readings per scientist and 
engineer observed in the general sujrvey of scientists and oigineers 
across nine fields of science. However, vrtien only persais engaged in 
research and developnent are included, the average number of journal 
article readings comes to 144, which is greater than the amount of reading 
across the nine fields of science. The amount of reading of technical 
reports is found to be lower for all professionals (44 readings per year) 
than for the scientists and engineers geierally (78 readings per year). 
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Table 1 

NisDber of RBadlngs of Journal Articles, Technical !)e|X)rts and Books 
by Professionals at Rockwell, Rocky Flats and ORI«i - 1983/84 



Journal Article Technicad Report Book 
Headings* Readings* Readings* 



^erage ^ual 

Readings 99 44 15 

Total Readings 

at 3 Sites** 514,000 228,000 78,000 



^Readings are defined as going ^^cnd the contents, title and abstract 
to the bod^ of the article, technical report or book. 

**Based on an estimate of 5,188 prof^sionals found at the three sites. 

SOURCE: King Research, Inc. - General Usage Survey (ns=137) 

However, the research and develc^roent energy professionals have an average 
of 69 readings per year. This seems to mean that non-R&D energy 
professionals do not read nearly as much in the journal literature as those 
engaged in R&D (67 versus 144 readings per year). This discrepancy in 
aroount of reading is even more pronounced in technical r^rt reading {69 
readings by R&D professionals, 26 readings by non-R&D professionals) . I^e 
number of books read by ncn-R&D professionals is about one-half the number 
read by those engaged in R&D (10 readings versus 21 readings) . 

It is estimated that the energy professionals ^>end an average of 
0.9 hours reading an article, 1.5 hours reading a technical r^rt and 4.4 
hours reading a book. This correspc»ids closely with the findings from the 
general ^BF survey of scientists and engineers. Professiaials' time is a 
scarce resource and they allocate their time to reflect how they can get 
the most return for their time invested. Thus, the fact that the energy 
professionals spend about 233 hours a year reading articles, tecJmical 
reports and books indicates that they consider the information read to be 
of significant value to them. In fact, their time ^ent reading is 
considered a component of what professionals are willing to pay for 
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Inforraation. If we use a typical salary or wage figure that includes other 
related direct costs*, it is estiaated that the average cost of reading per 
professional is $6,400 per year. Olie amount of time spent reading is 
sumarized in Table 2. The total cost of time qjent reading at the three 
locations is estimated to be about $33 million. 



Table 2 

Amount of Time Spent Reading and Cost of Reading Journal 

Articles, Technical Reports and Books Read by 
Professi<»ials at Rockwell, Rocky Plats and ORNL - 1983/84 



Journal 
Article 
Reading 



Technical 

R^rt 

Reading 



Book All 
Reading Reading 



Average 
Annual Time 
Spent Reading 

Average Cost 
of Time 
Spent Reading 

Total Cost of 
Time Spent 
lading 



95 hours 



$2,600 



$13.5 

milliwi 



75 hours 



$2,100 



$10.9 

million 



63 hours 



$1,700 



$8.8 

millicsi 



233 hours 



$6,400 



$33.3 

millicsi 



SOURCE: King Research, Inc. - General Usage Survey (n=137) 

T^e amount of time spent reading varies substantially by the work 
role of the professionals. 'Por exanple, the amount of time ^5ent by R&D 
professionals reading journal articles and technical reports is about 
double the time ^nt by non-R&D professionals. Results of the survey 
concerning tiiese two work roles are given in Table 3 on the next page. 



We also determined v^y the professionals read journal articles and 
technical reports. Ihe purposes of reading and the proportion of readings 
conducted for each purpose are sirnirarized in Table 4 on the next page. 



Salary of scientists and engineers is derived from the median R&D 
salary (Science Indicators 1982) extrapolated by the apprc^riate GNP 
Inplicit Price Deflator, plus 50% for fringe benefits and other related 
direct costs. The hourly equivalent to this salary is $27.38/hour. 
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Table 3 

Average Annual Time Spent Reading fcy Professionals Whose Primary 
Work Sole is Research and DeveloExnent and by Ml Other Professionals 
at Rocky Flats Technic2Ll Library Rockwell Ehergy Systons 
Groip Li±)rary, and ORIfii Library System - 1983/84 



Re^arch & Professionals In 
All Develc^mpnt All Other Prinary 

Professionals Professioneds Work Roles 



Journal Articles 


95 


hours 


126 hours 


73 


hours 


Technical Repor.ts 


75 


hours 


111 hours 


49 


hours 


Books 


63 


hours 


76 hours 


54 


hours 


Total Readings 


233 


hours 


313 hours 


176 


hours 



SOURCE: King Research, Inc. - General Usage Survey (n«137) 



Table 4 

Purposes of Reading and Prc^rtions of Readings of Journal 
Articles and Technical K^rts Read by Professionals at 
Rockwell, Rocky Plats and OPNL - 1983/84 



Purpose of 
Reading 


Journal 
Articles 


Technical 
Reports 


Findings Applied to Project 


54% 


64% 


Methods Applied to Project 


43 


50 


Research Proposal 


32 


18 


Prepare article, book or report 


39 


45 


Reference for article, etc. 


25 


45 


Professional develo^xoent 


82 


73 


Prepare lecture or presentation 


11 


18 


Planning, budgeting, and management 


18 


5 


Other 


4 


5 



SOCKCE: King Research, Inc. - General Usage Survey (n«137) 
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•n>ese purposes of reading by energy professionals correspond to results 
observed in the general survey of scientists and engineers. Results show 
that there are sever«d purposes for reading, but the most prevalent ones 
are for professional deveit^nient; to ^pply findings and methods to current 
projects; and to pr^re manuscripts for journal articles, technical 
r^rts or books. 

2.2 The Value of Q^ergy Intormation 

Obviously, since energy professionals spend &:> much time reading 
and for such useful purposes, they must be willing to pay an appropriate 
amount for the information, Ttm price they are willing to pay in addition 
to money paid for subscript icais, etc., includes their personal time and 
effort ej^sended to identify, gain access to and read the articles and 
technical reports. We have estimated this time and cost associated with 
readings of library materials. In addition, we asked the professionals to 
indicate what they would be willing to spend to obtain the article (or 
technical report), not including their investment of time. This price, 
which corresponds to ccaisianer surplus in eccnomic terms, is estimated to be 
56 per article reading and $56 per technical report reading.* The energy 
professionals did not have to pay the library for using their journals and 
tech;iical reports, so the price of the information material was not 
inclined in the calculations. Ihe estimates of how much users are willing 
to pay are given in Table 5. 

Ihe average vali«s in terms of willingness to pay are estimated to 
be about §36 for reading a journal article and about $102 for reading a 
technical report, /flying these estimates of value (e.g., willingness to 
pay) we arrive at a total of over $42 million. It is :ioted that these 
estimates of value of int ox-mat icn include only the cost (or price) borne by 
the user^. They do not include such library costs as purchase of sutscrip- 
tions, searches performed by reference librarians, etc. Ihe calculations 
of the average value per reading are subdivided by the appropriate methods 
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An outlier of $12,000 was excluded from the calculations. 
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Value of Information in Terns of What Users 
Are Willing to Pay - 1983/84 



ScHarce of 


"Willingness" 


Total 




Value Per Reading 


w 111 ingress value 


Value 




(S) 


($) 










Identification 




2.26 


$ 1.2 million 


Access 




3.41 


1.8 


Reading 




24.64 


12.7 


Additional Value 




6.00 


3.1 


Total 


$ 


36.31 


$18.8 million 


Techa 2^. Roadings 








Identification 


$ 


1.72 


$ 0.4 million 


^tocess 




3.65 


0.8 


Beading 




41.07 


9.4 


Additiaial Value 




5$. 00 


12.6 


Total 


$ 


102.44 


$23.4 million 



SOURa:: King Research, Inc. - General Usage Survey (n=137) 

of identification 9^- materials read. These methods included: by accident 
while browsing through the library mterials, from another person {i.e., a 
collGague), cited in an article or other publication, cited in a printed 
index, cited in the output of a coinputerized literature search, routed by 
the library, from a library accessions list and other (e.g., article sent 
by an author for review). Obviously, the time spent by the energy 
professionals for these methods varied substantially and therefore, was 
reflected in the estimates. 

Hie second level of value is measured in terms of the cwisequences 
that reading and using the information has on research, educaticsi and 
nttnagement. A portion of this higher order effect is estimated by asking 
the energy professionals to indicate if reading a ^)ecific recent article 
or technical report saved them or their co-workers any time on a current 
task or project. If yes, they were asked to estimate the approximate 
dollar value of the time saved. Ttiey were also aisked if there were dollar 
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saviDgs for other things such as ecjuiFnent and simply cx>8ts. Finally, they 
vjere asked to indicate the nuaber of co-workers involved in the savings and 
how nany of them also read the article or technical report. "Die indication 
of how many read the article or technical report was determined in order to 
adjust the estimates of total savings. For example, if two energy 
professionals on a project read an article, they both would have presunably 
achieved and reported the savings. Thus, the reported savings are halved. 
'!lie average savings found by reading journal articles is $385 and the 
average savings foind by reading tectviicjd reports is $706. Applying these 
average values to the amount of reading yields a total estimate of about 
$360 million value in terms of savings. !Ilie average and total savings are 
given in Table 6, 

Table 6 

Value of Info mat ion in Terms of 
Savings to User - 1983/84 



Source of 


Savings Value 


Total Savings 


Savings Value 


Per Reading 


Value 


Journal Readlnqs 






labor Savings 


§ 356 


$ 183 million 


Other Savings 






Total 


$ 385 


$ 198 milliwi 


Tech. I^. ReadirKis 






Labor Savings 


$ 590 


$ 135 million 


Other Savings 




2$ 


Total 


$ 706 


§ 161 millicxi 


Total 




$ 359 million 



SDURCE: King Research, Inc. ~ S^ial Usage Survey (n=B7) 

It is noted that the average nunber of people on a project who read 
the articles and technical reports was very close to two in each instance. 
Hius, the total estimated reported savings were $770 and $1,412 for journal 
article readings &nd technical report readings respectively. The 
corre^onding estimated savings in the 1981/82 energy study \7ere $590 and 
$1,280 respectively, which seem to be well within statistical limits. 
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SECTION 3 

TOE CSE AW) VSLUE OF INPOSMMTC^J ISfH 
IS ATTREBOTABLE TO LIBRARIES 



3.1 The Use of Libraries 

The libraries (Rocky Plats Technical Library, Rockwell Ehergy Sys- 
tems Groi?) Library and ORNL Library Systan^ are extensively used. In fact, 
the energy professionals at these locations are estimated to use the 
libraries (or theii services) an average of nearly 25 tinies per year. Uie 
libraries are used for a wide variety of purposes as indicated by the 
average annual uses below: 

• 21 times visited one the libraries 
a 17 times read materials in the library 

• 11 times checked materials out 

• 10 times photocopied mate * ials in the library 

• 8 times obtained assistance in locating materials fron a library 

staff person 

• 4 tl es obtained assistance in identifying relevant materials 

frcm a library staff member 

• 4 times used microform or microfiche equipnient 

• 2 times obtained materials frcm other libraries 

• Once did ^ search on library equipnent 

• Less than once obtained an oiJine bibliograj^iic search and used 
• idio visual equipment. 
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In terms of the reading of journal articles, technical reports and 
books, it is fouid that a substantial proportion of readings came fran 
library copies (see T&ble 7) . In fact, the estimated average annual ainount 
is 53 article readings per professional, 17 technical report readings per 
professional and 6 book readings per professional. The proportion of 
readings fron library loaterials observed in the general scientists and 
engineers survey was substantially lower for both articles (23%) and 
technical reports (21%). 

Table 7 

Amount of Reading, i-ran Library Copies of Journal Articles, Technical 
R^rts and Books: Rocky Flats Technical Library, Rockwell 
Ehergy Systems Groip Library and ORHi Library System - 1983/84 



Prcportion Average Annual Total Ainual 

of All Itoiber Nuirber of 

Readings of Readings/ Readings 
Professional 

53% 53 275,000 

39 17 88,000 

39 6 31,000 



Journal Articles 
Technical Reports 
Books 



SOURCE: King Research, Inc. - General Usage Survey (n=137) 

Professionals engaged in research and development read library 
materials substantially more frequently than non-R&D professiaials. Thus, 
the pattern in which there is more reading by R&D professionals carries 
into reading of library materials as well, but to an even greater degree. 
Carparisons of amount of reading of library materials are given in Table 8. 
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Table 8 

Average Annual Readings Frcni the Library Materials by Professionals Whose 
Priaary Work itole is Research and Development and All Other Professionals: 
Rocky Plats Techniced Library, Rockwell Ehergy systans 
Groip Library, and ORNL Library System - 1983/84 



Research & Professionals In 
All Developwsnt All Other Prijiary 

Professionals Professionals Work Roles 



Journal Articles 53 85 8 

Technical Reports 17 35 5 

Books 6 10 3 

Total Readings 76 130 16 



SOURCE; King Research, Inc. - General Usage Survey (n-137) 

Next we calculate the amount of value of information that is 
attributable to the three libraries. Then, in Section 4, we look into why 
the libraries are used so extensively. 

3.2 Value Attributable ho Libraries 

The estiniated value of energy related information was given in 
Section 2.2. For the value assessed in terms of user's willingness to pay, 
the estimated vcilue of an article reading is about $36 and for a technical 
report reading, about $102. For the value determined by savings resulting 
frcin reading, the value for an article reading is about $385 and $706 for a 
technical report reading. In this section we estimate the amount of this 
value that is attributable to the library materials and services. 



19 23 



Ttiete are estinated to be approximately 211,000 readings of journal 
articles and 87,000 readings of technical reports provided by the three 
libraries. Ohe corre^xxiding values of these readings calculated in terms 
of v^t users are willing to pay and in savings to the users are given in 
T^e 9 below. 

Table 9 

Willingness to Psry value and Savings Value 
Directly Attributable to Reading Library Journals and 
Technical Reports - 1983/84 

{$ million) 



Source of Value 


Willingness to Pay 


Savings Value 




Value 




Journal Readings 


$ 7.7 million 


$ 82 million 


Technical Beport Readings 


$ 8.9 million 


$ 62 million 


Total 


$16.6 million 


$ 144 million 



SOURCE: King Research, Inc. 



These, of course, are not the values that are found by 
substitutions to the use of library materials. Uider normal circumstances, 
the users could get these mterials from other sources. Thus, the values 
are determined by estimating the cost of using substitute sources for the 
information and determining the number of readings that would be lost if 
the total ejqjenditure levels were maintained. The values of the lost 
readings are yabat we consider to be the vsQues of the libraries. These 
values are given in Table 10. 
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Table 10 

Willingness to Pay Value and Savings value Found by Substituting 

for the Libraries: Rocky Plats Technical Library, Roclcwell 
Energy Systems Groi;p Library and QRNL Library systen - 1983/84 



($ millicn) 



Source of value 



Nutter of 

Lost 
Readings 



Willingner>s to Pay 
Value 



Savings Value 



Journal Readings 



47,600 



$1.7 millicm 



$ 18 mUlion 



Technical Report 
Readings 



13,600 



$ 1.4 niillion 



$ 10 million 



Total 



298,827 



$3.1 million 



$ 28 million 



SOURCE: King Research, Inc. 

An exanple is givei below to show hew the ccilculations are made. 
Appendix A gives further detail. 

•nie calculation of value begins with a determinaticxi of current 
costs of acquisition, technical processing, identification, access and 
reading of library materials. This is done for readings associated with 
various methods used to identify an article or technical report which has 
been read (such as found while browsing through a library journal or found 
in an online search output). Obviously, the costs for such methods of 
identification vary substantially. Ihe average cost of acquisition 
(including price and technical processing) is $3.94 per reading for journal 
articles (and $3.53 per reading for technical reports). The cost of 
identifying an article (which has been read) by means of online searching 
is $0.68, which includes searches performed by users (16%) and by reference 
librarians (84%). Access includes the labor cost of an average of eight 



ERIC 




minutes to go to the library and back and average cost of photocopying the 
article ($0.73) which happens about 30 percent of the time. IHie cost of 
reading an article is $24.64, so that the total cost of reading an article 
foiKd ty online searching is $33.64. This nunber is given in Table 11. 
The total cost of these readings is $536,000 ($33.64 x 15,932). 

^le 11 

Cost per Reading of Library Copies of Journal Articles and Technical 
Reports by Method of Identification: Rocky Flats Technical Library, 
Rockwell aiergy Systems Groip Library and OSUL Library f^stem - 1984 



Journal Articles 



Technical T^eports 



Source of 
Identifi- 
cation 



Total current Alternate 
Readings Cost Cost 
($) ($) 



Total Girrent Alternate 
Readings Cost Cost 
{$) ($) 



Browsing 


86,489 


$35.09 


$46.78 








Oolleegue 


18,208 


35.24 


42.91 


17,432 


$47.00 


$49.28 


Citation 


50,072 


35.24 


42.91 


26,147 


44.72 


49.28 


Printed 
Index 


27,312 


36.79 


49.01 


5,811 


48.97 


60.00 


Online 
Search 


15,932 


33.64 


42.68 


7,263 


45.31 


51.86 


Routing 


13,656 


34.30 


46.78 


8,716 


44.71 


51.86 


Lit. Acc. 

List 








18,884 


47.00 


51.86 


Other 








2,905 


44.72 


51.86 


Total 


211,669 






87,158 







SOURCE: King Research, Inc. 

If there was not a library available, an alternative source would 
have to be found for identifying and getting these articles in order to 
read then. Itie re^xsndents indicated that about 40 percent of the readings 
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involved tnique information (i.e., equally useful information was not known 
to exist elsewhere) . Of that infonoation icnown to exist elsevi^ere, 55 
percent of the re^xxidents indicated that it was accessible in another 
library, 18 percent fron\ a colleague, 18 percent from the author and nine 
percent from the publisher. However, it is noted that in order to 
duplicat<3 the access when identification is through browsing, another 
librf-y would have to oe anployed or the professional would have to 
subsc to the journal. 

The estinate of the alternative cost of getting journal articles 
through online searches is estiitated to be $42.68. If the budget is set, 
one could assune that only about 12,559 readings could be achieved through 
online searching ($536,000/$42.68) . Thus, about 3,373 readingL would be 
lost (15,932-12,559) . Ttie professionals spend about $28.48 in terms of 
their time getting the information and they indicate they would be willing 
to tpend $6.00 more for the article, if necessary. Hius, we get a total of 
$34.48 that they would be willing to pay for a reading. ^Therefore, the 
total value of the readings acconplished through browsing would yield a 
value in terms of willingness to pay of about $116,000 ($34.48 x 3,373). 
Oliis number is given in Table 12 along with similar values calculated for 
other methods of identification and for technical rQ>ort readings. The 
total value (willingness to pay) for jourrxal article readings is $1.7 
million and it is $1.4 million for technical r^rt readings. Together, 
the average willingness to pay value per energy professional per year is 
about $600. 

The savings value found by substitution materials and services is 
calculated in a similar miiner. For exanple, the 3,373 readings ideitified 
by online searching that would be lost if there was no local library 
available can be multiplied times the average savings per reading ($385) to 
get a savings value of $1.3 million. Table 13 gives the savings value 
^ich totals $18 million and $10 million for article and technical report 
readings re^)ectively. ^Iliis canes to an average savings value of $5,400 
per energy professional which is attributable to library materials and 
services. In a sense, this means that the library materials and services 
account for about ten percent of the productivity of the energy 
professionals. 
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T^le 12 



Total Library VSQue Found by Substitution for Library Services in 
Tenns of Willingness to Pay by Method of Identification: Rocky Flats 
Technic2d Library, Rockwell Energy Systects Groi^ Library and OBNL 

Library Systan - 1984 

($ millions) 



Method of 
Identification 



Journal Articles 



Technical Reports 



Browsing 

Cblleague 

Citation 

Printed 
Index 

online 
Search 

Routing 

Library 
Acc. List 

Other 



$0,787 
0.119 
0.327 

0.248 

0.116 
0.132 



$0,191 
0.216 

0.126 

0.124 
0.188 

0.322 
0.048 



Total 



$1,729 million 



$1,384 million 



SOURCE: King Research, Inc. 
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Table 13 



Ototal Library Value Found by Substitution for Library Services in 
Tentffl of Savings; Rocky Flats Technical Library, Rockwell Ehergy 
Systems Group Library and Q8NL Library System - 1984 



($ millions) 



Method of 

Identification Journal Articles Technical Reports 



Broi^sing 


$ 8.3 




colleague 


1.2 


$ 1.3 


Citation 


3.4 


2.7 


Printed Index 


2.6 


O.B 


Cliline Search 


1.3 


0.9 


Routing 


1.3 


1.3 


IJl>rari' Accession List 




2.2 


Other 




0.3 



Total $18.2 mUlion $ 9.5 inUlion 



SOURCE: King Research, Inc. 

Clearly, the savings 7alue is a "soft" estijnate, but probably is in the 
right order of magnitude. Even so, the extent of naaber of readings from 
library materials, time ^>ent ty energy professio.vils in reading these 
materials and the value determined by what they are willing to pay suggests 
that library materials and services are extremely valuable. 



SECTION 4 

PEKFOKM?^NC£ MS) EFFECTIVENESS OF LIBRARIES 



4.1 RRfl.qons for Using Libraries 

A quest icxi arises as to why there is so much use of libraries and 
so many readings of library materials. To begin with, awareness of the 
traditional library services is quite high, although analysis reveals that 
the level of awareness (and usage) of library services varies substantially 
anxxg specific services. All re^xxidents were aware that they cx)uld obtain 
assistance in locating materials from library staff, and nK>st (84%) had 
done so at least once. Almost 80 percent of the respondents to the 
question cn online bibliographic searching were aware that this service was 
available to then through the library, but only 45 percent had made use of 
this service. The overwhelming mjority of respondents (94%) were aware 
that the library could obtain materials for them frcxn other libraries, but 
28 percent of those aware had never done so. However, although all but 
about three percent were aware that they could obtain assistance In iden- 
tifying relevant materials frcm a library staff perscsi, about one-fourth 
l^d never done so. About 36 percent of those responding were unaware that 
the libraries offered audiovisual equipment other than microform equipment. 
Nearly half of those re^xxiding were aware of this service, but had never 
used it. 

Examination of the survey responses reveals, not surprisingly, that 
the most "conventional" library services are used more often than those 
that are generally considered less conventional or traditional. For 
example, personal visits to the library, reading materials in the library, 
photocopying materials in the library, checking out materials, and 
obtaining assistance in locating materials were performed an average of 21, 
17, 10, 11, and 8 times respectively in the last year by respondents. On 
the other hand, re^xandents used microform equipment an average of fewer 
than 4 times in the last year, used both other audiovisual equipment and 
had an online bibliographic search performed by them by the library an 
average of less than once during the past year, and performed a search on 



library equipnient «n average of less than once. Interlibrary loan, a 
fairly conventional service, was performed by the library an average of 
only twice in the past year per pre 'Sessional. This, however, nay possibly 
be due to the sufficiency of the library's own collect icji for the majority 
of needed it«ns, and %*hen multiplied by the nianber of patrons, results in a 
sizable volume of interlibrary borrowing. Another conventional service, 
obtaining assistance in identifying relevant mterials, was carried out by 
respondents cn an average of four or so times in the last year. More than 
25 percent of the respondents were aware of this service. 

Another factor that enters into high library use and readings of 
library naterials is that a high proportion of the professionals at the 
three locations are located near a library. In fact, over 60 percent of 
the professionals are within five minutes of their library and over 75 
percent within ten minutes. The significance of the distance (in time) is 
portrayed in Figure 1. It is shown that energy professionals who are less 
than five lainutes to the library average 8.8 visits a month to the library. 
When the energy professionals are located over ten minutes away, the 
average number of times they visit the library drops to below once a month. 
However, the number of uses of the library per month only decreases frcxn 
about seven uses per month for those under five mijiutes away to about four 
uses for those over ten minutes away. 

Materials read in libraries are, on the average, quite a bit older 
than materials read from other sources. This suggests that energy 
professionals rely on their libraries to gain access to older articles and 
technical r^rts. The age of readings is given in Table 14 for journal 
articles and technical reports. 

The average age of jouii^. ^ ' fron library copies is 3.5 

years coipared to less than a year toi uil. e- read from personal sub- 
scriptions and 1.5 years for articles obtained frcn oUier sources. Similar- 
ly, the average age of technical reports read frar. library copies, 2.4 
years, is double that of technical reports read from other -purees (1.2 
years) . 
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Figure 1 . Ntiiter of Visits to the Ubrary Per ftntfi as a Rnction of Dlstcnce 
to tl« Ubrary in Minutes 



Table 14 

Proportion of Readings in Libraries and Readings 
from Other Sources by ^^e of Publications - May 1S84 

(%) 







•TfMimAl Articles^ 




•ferfmical Rfioorts** 


Year 


Library Personal 
Copy Subscriptions 


Other 


Library 

Ocspy 


Other 


1984 


41% 


84% 


69% 


50% 




1983 


16 


10 


13 


19 


30 


1982 


8 


3 


6 


0 


8 


1977-81 


22 


1 


13 


19 


13 


1972-76 


5 


1 


6 


13 


2 


1962-71 


5 


1 


0 


0 


0 


<1962 


3 


0 


0 


0 


0 



Average 3.5 0.6 1.5 2.4 1.2 

Age years years years years years 

SOURCE: King Research, Inc. - Statistical Indicators Survey {n*=195, 
n**«664) 

Another factor which contributes to the reading of library materi- 
als has to do with the nunber of journal subscriptions taken by the energy 
professionals. If the professionals do not subscribe to many journals, 
th«^ are more likely to get access to their frequently read journals from 
other sources. For example, scientists were asked to indicate v^ether 
there are any journals frctn which they read at least ten articles per year 
to which they ifc-HQt personally subscribe. Of those vAio said yes {about 
cne-talf), the average nunber of subscriptions received by them %#as 3.3 
conpared to 5.8 subscriptions for those who said no. Of the subscriptions 
received, about two-thirds were paid for by the scientist. Olie renaining 
were mostly paid by their en?>loyers. The relationship here was about two 
to one for yes to no. These results are displayed in Table 15. 
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Table 15 

Average Nuuber of Subscriptions for Scientists 
WtK> Do and Do Not Gain Access to Frequently Read Journals 
(Over Ten Articles Per Year) - 1984 



Scientist Head at Least Ten Articles 
Per Year Prom Journals to Whijh 
He Does Not Personally Subscribe 



Yes 



No 



Average Total 

Persc«\al 

Subscriptions 



3.3 subscriptions 



Average Subscriptions 2.3 subscriptions 
Paid By Scientist 



5.8 subscriptions 
4.5 subscriptions 



SOURCE: King Research, Inc. - Statistical Indicators Survey (n=167) 

Of the scientists who frequently read journals to \^*iich they do not 
subscribe, most of them (76%) get than from the library. As shown in Table 
16, the reasons for not subscribing to the journal are the price of the 
journal (66%) and because it is readily available from the library (61%) . 
Thus, there appears to be an economic reason for the high level of reading 
library materials. 
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Table 16 

Proportion of Scientists Who Frequently Read Journals to 
Which They Do Not Siisscribe By Sources of These Journals 
and Reasons for Not Personally Subscribing to Journal - 1984 



Proportion of 
Scientists (%) 



Sources of Journals 

Borrow from Colleague 21% 

Library Copy 76 
Other 3 

RftaftnnR for Mnh PPrsonallv 
Subscribing to Journal 

Readily Available from Colleague 10%* 

Price of Journal 66 

Readily Available from Library 61 
Other 3 



*Prc^rtions do not add to 100% since respondents could have more than 
one rezison. 

SOURCE: King Research, Inc. - Statistical Indicators Survey Cn=167) 
4.2 library Performance 

Cne a;^>ect of library performance deals with the output of services 
provided. Hhis output has been given in terms of amount of materials pro- 
vided, nurtoer of searches performed and so on as part of the value calculus 
in the previous sections. Other measures of performance of the libraries 
include timeliness and quality. Ilhe timeliness of online and manual 
searching is measured by subtracting the time necessary to get the search 
output from the required amount of time to get it. In about 78 percent of 
the searches the output was delivered on time or ahead of the required 
time. Thus, about one-fifth of the search results were delivered after the 
required time. The average amount of time of late deliveries, for those 
delivered after the required time, was about 6 days. Quality is assessed 
below from the user's per^>ective of their satisfaction with various 
library services. 
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4.3 Library Effectiveness 



Library effectiveness is determined from the per^)ective of the 
library users. It includes such measures as amount of use and user 
satisfaction with the services. WG have discussed amount of library use in 
previous sections so that it is not repeated here. However, user satisfac- 
tion was obtained in the general, usage survey, Qiergy professionals were 
asked to rate their general satisfaction with library services. Only 
library users were asked to respond to these questions. A sunmary of 
satisfaction with specific library services are given below in Table 17. 



Table 17 

Proportion of Professionals Who Indicated Various 
Degrees of Satisfaction With Library Services; Rocky Flats 
Technical Library, Itockwell ES^ergy Systems Groip Library, 
and QRNL Library System - 1984 



(1) (2) (3) (4) (5) 

Very Dis- Neither . Satis- Very Average 
dis- satis- satis- fied satis- Satis- 
sat is- fied fied fied faction 
fied nor dis- Score 

satis- 
fied 

(%) {%) {%) (%) (%) 



Assistance 
in acquiring 

naterials 0 0 9 30 61 4,52 

Assistance 
in locating 
naterials from 
a library staff 

meinber 1 0 5 29 64 4.52 

Assistance in 
in searching 
for relevant 
materials fcy 

a library 

staff rnanber C 0 8 39 54 4.50 
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Table 17 (Continued) 





(1) 


(2) 


(3) 


(4) 


(5) 






Very 

UO-S— 

satis- 
fied 


Dis- 
fied 


Neither 
fied 

nor dis- 
satis- 
fied 


Satis- 
fied 


Very 

satis— 

fied 


Average 
Satis- 
factic*i 
Score 






(%) 


(%) 


(%> 


(%) 




Microform 

reading/ 

printing 

equipment 

available 


3 


3 


20 


41 


33 


3.98 



Ctollecticn of 
inc^ograp^, 
r^>orts^ 
journals 
available for 
reading in the 

library 0 6 12 40 43 

Staff Assistance in Locating and Obtaining Materials 



Level of library 

staff knowledge 0 0 

S|)eed of providing 
assistance 0 0 

^jeed of aoquirii.g 

naterials 2 6 

online Bibliographic Searching 

Relevance of out- 
put to your 

infctmation needs 0 2 

Nmber of 
references in 

search output 0 0 

Time between your 

request and your 

receipt of search 

output 0 0 



8 



22 



23 



17 



42 



41 



40 



41 



50 



54 



52 



29 



34 



33 



4.23 



4.50 
4.48 

3.85 



4.07 



4.16 



52 



39 



4.30 
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Table 17 (C3ontinued) 



(1) 


(2) 


(3) 


(4) 


(5) 




Very 


Dis- 


Neither 


Satis- 


Very 


Average 


dis- 


satis- 


satis- 


fied 


satis— 


Satis— 


satis- 


fied 


fied 




fied 


faction 


fied 




nor dis- 






Score 






satis- 












fied 








(%) 


(%) 


(%) 


(%) 







Searcher ' s 
technical 
performance 
(facility with 
and knowledge 
of search 
techniques and 
vocabulary) 

Searcher ' s 
knowledge of 
subject field 
searched 

Cost of having 
search performed 



45 



50 



4.45 



26 



46 



47 



33 



28 



17 



4.06 



3.59 



(Staining Materials fron Other Libraries {e.g., interlibrary loan) 

Time interval 
between your 
request of 
the iTQterials 
and your 

receiving them 2 5 



28 



45 



22 



3.86 



Cost of detain- 
ing materials 
from other 
libraries 



24 



45 



26 



3.91 



Packet of Materials Prc^red for you by a Library Staff Hanber cn a 
Specific Subject 



Ntanber of iters 
in the packet 



12 



65 



24 



4.16 



34 



43 



Table 17 (Continued) 





(1) 


(2) 


(3) 


(4) 


(5) 






Very 
dis- 
satis- 
fied 


Dis- 
satis- 
fied 


Neither 
satis- 
fied 

nor dis- 
satis- 
fied 


Satis- 
fied 


Very 
satis- 
fied 


Average 
satis- 
faction 
Score 




(%) 


{%) 


(%) 


(%) 


(%) 




Relevance of 
items in packet 
to your infor- 
mation needs 


0 


0 


6 


72 


22 


4.16 


CXirrentness of 
it€!ns in tiie 
packet 


0 


6 


6 


50 


39 


4.25 


£^}eed at which 
the packet was 
pr^jared 


0 


0 


22 


33 


44 


4.18 



SOURCE: King Research, Inc., General Usage Survey {n-137) 



Satisfaction with specific library services, was generally fairly 
high. Average satisfaction scores ranged fzcm a lew of 3.59 (cost of 
havijig search perfomied) to a high of 4.52 for ascistemce in (1) acquiring 
and (2) locating mterials. However, a small prc^x)rtion of respondents 
noted sane dissatisfaction with assistance in locating materials (1%), 
microform reading/printing equipnent available (6%), and the collection of 
monographs, reports, and journals available for reading in the library 
(6%) , respectively. Not surprisingly, the highest percentage of "less than 
satisfied" respcr.ses (just over 26%) fell into the category of microform 
reading/printing equipnent. It should be noted, however, that nearly 18 
percent of the responses dealing with collection of monographs, technical 
reports, journals, etc., also falls into the "less than satisfied" 
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category. In contrast, none of the respondents who had used the services 
in the past year rated then^elves as dissatisfied or very dissatisfied with 
assistance in acquiring mterials and with assistance in searching for 
relevant materials. 

Satisfaction with staff assistance in locating and obtaining 
materials was also fairly high. However, the responses to speed of 
acquiring materials shew the highest level of "less than satisfied", with 
two percent of respondents feeling very dissatisfied, six percent feeling 
dissatisfied, and 22 percait neither satisfied nor dissatisfied. 

Levels of satisfaction with online bibliographic searching 
performance attributes reveal that the majority of those respondents for 
v?hoin the library had performed a search in the past year felt satisfied or 
better with the performance attributes of relevance of output, nwtei of 
references in search output, time between request and receipt of search 
output, searcher's technical performance and searcher's knowledge of 
subject field searched. However, only 50 percent of those responding in 
terms of a le^;el of satisfaction with the cost of having search performed 
felt satisfied or better, with nearly 46 percent feeling neither satisfied 
nor dissatisfied. Although the majority of respondents (74%) felt 
satisfied or better about the searcher's knowledge of the subject field 
searched, another 26 percent were neither satisfied nor dissatisfied with 
this performance attribute. 

Levels of satisfaction with performance attributes associated with 
obtaining materials from other libraries indicate that the majority ot 
respondents felt satisfied or better (66%, 71%) with the time interval 
between their request of materials and their receiving them, and with the 
cost of obtaining materials from other libraries. Interestingly enough, 
about one quarter of the re^xxidents (28% and 24%) felt neither satisfied 
nor dissatisfied about the time interval and the cost involved in obtaining 
materials from other libraries. 




Satisfaction with performance attributes associated with packets of 
materiads prepared by library staff on a particular subject is generally 
hi^. However, it should be noted that the ninber of respondents actually 
rating their level of satisfaction with packets of naterials is quite lew, 
as the number of respondents \A)o had not used this service within the last 
year was miformly high. 'Aiis low degree of usage mig^it bear further 
investigation as one could hypothesize that respondents who indicated they 
had not used the service may not have been aware of its availability. For 
only one performance attribute, currentness cf itemas in the packet, c3o 
those respondents \^ have received a packet express dissatisfaction - not 
quite six percent were dissatisfied with the currentness of itenis in the 
packet. 

V4ien asked to rate 'che library's collection, 73 percaA of those 
doing so felt that the collection was coiplete enough for their needs, with 
the remainifig 28 percent rating the collection as nsL ccxiplete enough for 
their needs. Hiose re^xxidents relying in the negative made ccnroents on 
the library collection's ^rtcoming that were diverse and fairly specific, 
ranging from coninents like: "need more materials on subject x", "library's 
X are not current", to "materials I need always seem to be on loan and not 
available". Some respondents also took the time to include positive 
coiiiTients anchor to indicate satisfaction with certain types of library 
naterials in this area. 

Re^xxises to the question which deals with whether or not respon- 
dents have received unsolicited materials from the library and asks them to 
rate the level of usefulness of these materials, are noteworthy in that the 
majority of respondents consistently indicate that they had not received 
these unsolicited materials. (Wiether or not this is indeed the case, or 
whether a higher proportion of re^jondents had indeed received unsolicited 
nateriaas from the library but did not realize this fact can not be 
ascertained.) Of all three types of unsolicited materials, only general 
information was received by more than 10 percent {36%) of those responding. 
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It is noted that ratings of level of usefulness given here should 
be treated with caution. Of those that had received unsolicited goieral 
information, 88 percent rated it as useful, and six percent as not «iseful. 
Those v?ho had received an unsolicited literature search differed as to its 
usefulness; 22 percent rated it as not useful and 78 percent as useful. On 
the other hand, all of those receiving objective analyses of issues of 
current high interest rated them as useful. 

Responses to library usage appear to indicate an acceptable level 
of satisfaction, and a fairly high volume of usage for "conventional" 
services, etc., for the most part. However, one might hypothesize that 
library patron level of awareness, both of the availability of certain 
specific library services, and of when they are actually receiving library 
services/mterials (especially those that are unsolicited) may well need 
improvement. 
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SECTION 5 

RELATIONSHIP OP PE31EORMANCE TO EFFECTIVENESS TO VALUE 



5.1 Introduction 

An attempt is made in this section to relate some librarian 
capabilities to their performance, performance to effectiveness, and 
performance to value. All of the analysis deals with online and manual 
bibliographic searching. The librarian capabilities investigated include 
their knowledge of the subject in ^ich specific searches are performed and 
their technical searching skills (i.e., facility with and knowledge of 
search techniques and vocabulary) . Performance is measured by timeliness 
of re^xDnse to search, relevance of outpnit to information needs and nuinber 
of references in search output. Effectiveness is measured in terms of user 
satisfaction and number of searches performed per month. These 
relationships are given in Section 5,2 and 5.3. In Section 5.4, a 
trade-off analysis is performed in which user time (i*e., willingness to 
pay value) is determined as a function of response time and quality of 
search in terms of relevance of output. 

5.2 Relationship of Li brarian Capabilit ies to Performance 

The first correlation is done between ranking of the searcher s 
knowledge of the subject field searched and satisfaction with relevance of 
output. Ilie searcher's qualification in terms of knowledge of the subject 
field searched is found by having the user rank this knowledge from not 
well qualified (rank of 1) to very well qualified (rank of 5). This 
ranking was done for a specific search. The users were also asked to 
indicate their degree of satisfaction with the relevance of items 
identified to their informaticsi needs.* This relationship between the 
capability and performance is depicted in Figure 2 ^ich shows that 
rankings of the searcher's knowledge at level two, on the average, yields a 
dissatisfaction with the search (i.e., satisfaction score of two). 



* Ftossibly a better means of observing the capabilities and performance 
would be an expert in the field and not the user. However, there are 
arguments for both methods. 
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Figure 2. Relation of Search Rerfonmice as a Finctlon of the Searcher's 
KJiowledge of Sii^ject Field 



However, as the r2mking of searcher's knowledge increases, the level of 

satisfaction increases to the point where a ranking of searcher s 

qualification in terms of subject knowledge of five yields an average 
satisfaction score of 4.8. 

Similarly, higher rankings of searcher's technical skills yield 
higlier satisfaction with relevance of output (Figure 3) . However, the 
levels of satisfaction with relevance of output are generally not as high 
as those observed with the searcher s knowledge of the subject field 
searched. Ihus, one would conclude that the s^rcher s subject knowledge 
is probably more inportant than the searcher s technical skill. 

5.3 Relationship of Performance to Effectiveness 

Another important relationship is between speed of delivery 
(performance) and satisfaction with speed of delivery (effectiveness) . Oie 
chart in Figure 4 shows that energy professionals are dissatisfied (2.0) if 
the search results are delivered more than five days beyond the required 
time indicated by the user. Oi the average, they are satisfied (4.2) if 
the delivery is not over five days beyond the required time. If the 
results are delivered ahead o> the required time, the users, on the 
average, are close to being very satisfied (4.8). 

In this section, we also show that amount of use of library 
services is related to individual measures of performance (specified by the 
users) . The measures of performance include satisfaction with relevance of 
output, number of references retrieved and timeliness of re^xxises. First 
we show that average ntnnber of online searches performed per year for users 
is related to satisfaction with relevance of output. The chart in Figure 5 
shows that users \iho are dissatisfied with relevance of search output, on 
the average, ask for searches only 1 1/2 times per year. Vftiereas users who 
are very satisfied with relevance of output ask for searches an average of 
3.2 times per year. Referring back to the searcher s knowledge of the 
subject field searched, a low ranking (2) resulted in the user being 
dissatisfied and a high ranking (5) yielded a score close to being very 
satisfitK3. !Ihus, one might convert lew searcher knowledge to 1.5 searches 
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Figure 3. Relation of Search PerfonnaTce as a Firctlon of the Searcher's 
TK^nical Skill 
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Figure 4, Relation of Search Effectiveness as a Fimicn of Speed of 
Delivery 
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Figure 5. Relcjticn of Nuiter of Searches PerfoniEd as a Function of 
Perfommce in Tenrs of RBlevax:e of CXJtout 
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per year and high searcher kxiowledge to about 3.0 searches per year, "rtiese 
nimbersr in turn, could be converted to value since fewer (or inore) 
searches directly affect use of priroary ii'*foniBtion, and hence, value. 

Similarly, the average number of online searches requested per year 
is related to satisfaction with nuniber of references ret.rieved (Figure 6) 
and satisfaction with timeliness of search response (Figure 7) . However, 
these relationships do not appear to be as strong as the relationship of 
niitfcer of searches performed and relevance of search output. For e'tanple, 
the average nuitber of searches performed per year for those v^o are 
generally dissatisfied and for those who are very satisfied is 2.0 searches 
per: year and 2.8 searches per year respectively, thus showing a difference 
in the average of less than one search per year. As one would e^qject, the 
average number of searches performed per year for those who are neither 
satisfied noi dissatisfied (2.0 searches per year) is lower than the nirober 
perfonned per year for those vdio are very satisfied (3.2 searches per 
year) , 

5.4 Tradeoff of Performance and Value 

In the previous sections we were able to demonstrate a relationship 
between nunber of searches requested and, independently, performance 
attributes such as timeliness and quality of online searching. The problem 
is that caie does not know the relative cent ribut ion these perfomiance 
attributes have vrfien considered together. In this section we give an 
analysis of how timeliness and quality together affect the vadue of search- 
ing eaqjressed in terms of user time- (i.e., their willingness to pay). Ohe 
analysis is performed using a statistical method called conjoint analysis. 
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Figure 6. ReliTJon of Senrches RBrformed os a Rjxtion of Performnce in 
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Figure 7. Relation of Jiirter of Searches RerfcnEd as a Rrction of Performnce 
in Terns of Tineliness 
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Oonjoint analysis is a marketing research tool developed to aid 
system nanagers in estiioating the relative inportance o£ niinber of system 
performance attributes. Wiis method requires potential users to make 
overall -^udgenients about a set of cocplex alternatives involving different 
oocibinations of the aystem performance attributes, ^is is done by having 
the users rank their preference for the various oonbinations of system 
performance attributes, "fliese rankings are then used to arrive at a set of 
utility scales for the system attributes that are ccrpatible with original 
overall rankings. 

In our particular situation , there are three factors that are 
expected to influence users' preference: price (in tenns of user time), 
quality of search, and speed of resnc^^se- Here we consider three levels of 
price (less than a half hour, between 1/2 and 1 hour, and between 1 and 2 
hours) emd quality of search (high relevance, medium relevance and low 
relevance of items retrieved) . iSiree different speeds of losponse are also 
considered (within a day, between 1 and 3 days, and over three days) . 
Value is determined by what users are willing to i y in terms of their 
time. Since it has been shown that the value of library services is 
partially determined by saving user's time (a scarce resource), the survey 
respondents were asked to think of the "or ice" paid for searches performed 
for them in terms of their time. 

The data collection for conjoint analysis requires that 
professionals indicate the relative importance of the "price" and 
performance attributes by comparing pairs of the "price" and attributes, 
one at a time. Each pairwise cwiparison focuses on two of the three 
factors and, in effect, ignores the remaining one factor by not specifying 
it. Table 18 sunmarizes the alternatives or levels for each of these three 
factors. 



48 



5/ 



Table 18 

Performance Attributes, "Price" and ^5ieir Levels 



Performance Attributes ^stSL PeSgCigtion 



Quality of Search 


1 


not ^)ecified* 




2 


high relevance of items 




3 


medium relevance of items 




4 


low relevance of items 


^)eed of HesponBd 


1 


not specified* 




2 


within 1 




3 


between 1 and 3 days 




4 


over three days 


"Price" in terms of User e Time 


1 


not specified* 




2 


1/2 hour 




3 


1/2 to 1 hour 




4 


1 to 2 hours 



Not specified ♦neans that the factor is not one of the two factors 
included in the pairwise ccnparison. 



Next, we sunnarize the survey results for rankings mde by users. 
Il^e mean (or average) rank, as well as the minimun and maximum ranks for 
each pairwise comparison, is presented. If the mean rankings are arranged 
in order, we have a reasonable impression of how most re^ondents cxxpleted 
their questionnaires. Hie minimun and naximim values indicate tlie ^read 
in their re^xxises. 
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In the imtrix below, we give performance attributes: quality of 
search and "price" in terms of user time. The professicmals ranked the 
nine conljinaticns of levels irom 1 to 9. Presiaiably, the highest rank (1) 
is high relevance at a "price" of only 1/2 hour. The lowest rank (9) is 
lew relevance at a cost of 1-2 hours, Ttie regaining cells were ranked from 
2 to 8. In the matrix we show the average ranked value and minimum and 
maximum ranks observed in the survey. 

Quality gf Search "Price" in Terms of User Time 



1/2 hour 1/2-1 hour 1-2 hours 









, - 


High relevance of items 


1 


2.10 


3.63 




{1, 1) 


(2, 4) 


(3r 7) 


Mediun relevance of items 


3.73 


4.95 


6.33 




(2, 6) 


(4, 6) 


(4, 8) 


Low relevance of items 


6.50 


7.78 


9 




(3, 7) 


(6, 8) 


(9, 9) 



Generally, users ranked second the cell of high relevance and 
"price" at 1/2 hour to 1 hour. This suggests that high relevance is more 
inportant than the price paid in terms of the user's time. However, at the 
price of 1 to 2 hours about one-half of the respondents preferred lower 
relevance (medium). This is where the price becomes more iiif»rtant than 
relevance. 

Cn the next page are two other attributes: speed of response and 
price in terms of user time. The cells are also ranked 1 to 9. Again, 
average ranks (and minimum and maximum) are presented- 



Speed P£ ReSPCflSe Price in Terms of User Time 



1/2 hour 1/2-1 hour 1-2 hours 



fv <^ iJciy 


2^ 


2.75 


4.97 




(Ir 1) 


(2, 4) 


(3, 7) 


K/in 3 days 


2.97 


4.81 


7.06 




(2, 4) 


(3, 6) 


(6, 8) 


Over 3 days 


5.25 


7.19 


9 




{3, 7) 


(6, 8) 


(9, 9) 



Here the second choice was clearly ^lit between (1) the choice of 
speed within the day and "price" at 1/2 to 1 hour and (2) the choice of 
speed within three days and "price" at 3/2 hour. TSius, speed does not 
appear to be as inportant as cotpared to relevance of search items when 
users are forced to choose betwten these perfomance attributes emd price 
in terrns of user time. This premise is confirmed below when users are 
required to make choices between levels of quality of search and speed of 
response. The tendency is to choose quality over speed of response. 

Below are the results of the last combination of pairs of 
attributes: quality of search and speed of response. 

SPee<3 P£ Response Quality of Search 





High 


Medium 


Low 




Relevance 


Relevance 


Relevance 


H/in Day 


1 


3.49 


- ■-■ - 
4.97 




(1, 1) 


(2, 4) 


(3, 7) 


H^in 3 days 


2.97 


4.81 


7.06 




(2, 4) 


{3, 6) 


(6, 8) 


Over 3 days 


5.25 


7.19 


9 




(3, 7) 


(6, 8) 


(9, 9) 

fa 1 «^ 
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The survey results were then used to estiaate utility scales for 
the value and perfonnance attributes. These utility values are given in 
!2^1es 19 and 20. 



T^le 19 

Normalized Utilities f rem Conjoint Analysis 
By Perfonnance Attributes for Three Librari^ - 1984 



(MANUAL SEAF 



Performance 
Attribute 



Level 



One 



Library 



Two 



Three 



Quality of Search 



^Jeed of Response 



Price (in Terms 
of User's Time) 



1. not specified 


2 


8 


12 


2. high relevance 


37 


40 


43 


3. rosdiian relevance 


16 


24 


22 


4. low relevance 


0 


0 


0 


1. not specified 


n 


0 


0 


2, within 1 ^y 


32 


26 


24 


3. between 1-3 days 


16 


15 


5 


4. over 3 days 


0 


0 


0 


1. not specified 


0 


4 


6 


2. 1/2 hour 


30 


33 


33 


3. 1/2 to 1 hour 


14 


22 


13 


4. 1 to 2 hours 


3 


0 


0 



SOURCE: King Research, Inc. 
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Table 20 

Nomalized Utilities front Conjoint Analysis 
By Perfontance Attributes for Three Libraries - 1984 

{CMJm SEARCEES) 



Library 

Performance 

Attribute Level One TVw) Three 



Quality of Search 


1. 


not i^)ecified 


4 


4 


7 




2. 


high relevance 


39 


39 


43 




3. 


medium relevance 


20 


21 


22 




4. 


low relevance 


0 


0 


0 


S^>eed of Besponse 


1. 


not specified 


0 


0 


0 




2. 


within 1 day 


31 


33 


28 




3. 


between 1-3 days 


16 


16 


16 




4. 


over 3 days 


0 


0 


0 


Price (in Terms 


1. 


not specified 


0 


0 


0 


of User's Time) 


2. 


1/2 hour 


29 


28 


28 




3. 


1/2 to 1 hour 


16 


17 


16 




4, 


1 to 2 hours 


1 


4 


0 



SOURCE: King Research, Inc. 



Several interesting results stem from the utilities given in Tables 
19 and 20. The factor with the greatest utility and therefore the greatest 
iiTiportance is quality of search. T^e renaming two factors "price" and 
speed, have similar utilities about 3/4 as high as quality. 
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Tbe total utility of a service or a conbinaticn of performance 
attributes can be calculated by adding the appropriate utility associated 
with the level of the attributes of that service. For example, si^spose a 
mnual search service at library One had quality of search at medium 
relevance, and speed of response at over three days, and a price (in terms 
of user's time) of 1/2 hour. The total utility would be 46 (16-K)+30) . 
Ototal utilities for other conbinations of quality, speed, and "price" can 
be ccnputed similarly. Indeed, the total utility can be cocputed for 
combinations not appearing in last two matrices. 

It should be mentioned that the range of utilities covered by 
Tables 19 and 20 depend on the £5)ecific factors and levels included in the 
questionnaire. If additional factors, or additi<»ial levels within one of 
the factors had been included, the set of utility coupcnents would change. 
For example, if the quality of reproduction had beed included as a third 
performance attribute, the set of carponent utilities of T^les 19 and 20 
would change. Such changes could affect the relative iiiportance of the 
factors. 



One can also directly convert "price" (value in terms of 
willingness to pay) into dollars and coipare combinations of the qiiality of 
timeliness against the value or dollar amount. For example, the utilities 
for value (i.e., price in terms of user time) is given below for manual 
searches and online searches. The conversion factor is about $0.90 per 
utility. Thus, for nanual searching the value of one day response and high 
level of relevance is $59.40. For 2 1/2 days response and medium level of 
relevance the value is $35.10, and so on. The response curves for the 
values are shown in Figures 8 and 9. These are for all libraries. 

One can see in Figure 8 that users are willing to pay $59 for 
searches that have high relevance of respwises and are delivered in one 
day. If we lower the quality to medium relevance of re^xxises, the users 
are willing to pay only $45. If the speed of delivery of these searches is 
increased to over three days, the users are willing to pay $23 less ($45 
minus $22) . Generally, users are willing to pay more for online searches 
than manual searches, except when they are of low quality or slow re^x)nse 
tiroes. It is felt that this model provides a valuable tool for decision- 
making in a library or in designing a search system. 
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Figure 8: Value of Manual Searches to Relationship of Level 
of Relevance and Speed of Response 
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1 Valu* to Dfttrs 

In Terms of Their Time 
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Figure 9: Relationship of Value of Online Searches to Level 
of Relevance and Speed of Response 



AFPQOIX A 
I^AILES COST ANALYSIS 



Uie current cost of journal reading is sucinarized in Table A.l below. 



Table A.l 

Current Cost of Reading Journal Articles 
by Method of Identification and Activity - 1984 



Total 

Activity Cost 



Method of Nurober of ($ mil- 

Identificaticn Readings Stor^e identification Access Reading lion) 

Source of Cost 



Bi.G%trsin9 86,489 

Professionals $ 5.78 $24.64 $2,631 

Other $3.94 $0.73 0.404 

Colleague 18,208 

Professionals 2.28 3.65 24.64 0.557 

Other 3.94 0.73 0.087 

Cited in Article 50.072 

Professionals 2.28 3.65 24.64 1.531 

Other 3.94 0.73 0.234 

Printed Index 27,312 

Professionals 2.21 3.65 24.64 0.833 

Other 3.94 1.62 0.73 0.172 

Qiline Search 15,932 

Professionals 0,19 3.65 24.64 0.454 

Other 3.94 0.49 0.73 0.082 

Routed 13,656 

Professionals 4.56 3.65 24.64 0.411 

Other 3.94 0.73 0.05/ 



SOURCS:: King Research, inc. 
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The principal assus^^tions leading to the results above are as 

follows: 



a. Browsing . It was estimated from the survey that the average 
time taken to get to a library is 4 minutes. "Rius, the total 
time would be 8 minutes. It is assumed that three articles are 
read per visit and that it takes 2ibou. 10 minutes of browsing 
per article read in this imnner. It was estimated from the 
survey that 29 percent of the articles read in libraries are 
photo- copied. 'Then the average cost would be about $0.73 per 
article read ($0.25 x ten pages x 0.29). 

b. Qplleaaue . It is assumed that identification of articles from 
a colleague would require about 5 minutes and the article copy 
would be obtained from the library (8-minute trips and photo- 
copy at $0.73) . 

c. Cited in Article . No time or cost is allocated to identifica- 
tion of the article since reading an article is accounted for 
elsewhere. Ac:cess is as above. 

d. Search of Printed Index . It was estimated from the survey that 
38 percent of the manual searches were performed by the users. 
Anount of time spent £;earching was determined frcm the NSF 
survey. Hie time and cost of librarian searches was adopted 
frcm the EEB value re^arch (p. 66) . It was assuned that users 
also spent 5 minutes in these searches. The users average 5.0 
article readings per search, thus costs were divided by this 
nuirber. Access is the same as above. 

e. Online Search , 'flie survey indicated that users performed about 
16 percent of the online searches themselves, llieir time was 
derived from the NB? survey and other search costs were found 
from the EIB study (p, 68). Users averaged 19.3 articles read 
per search. Access is the same as above. 

g. Routed Journals . Here identification time was assumed to be 10 
minutes and access time assumed to be 2 minutes. 



The reading time was determined frcxn the user surveys. 




58 



67 



Table A. 2 

Alternative C3ost of Reading Journal Articles 
by Method of Identification and itotivity - 1984 



Hethod of 

Identification Identification Access Heading All Costs New Readings 

{$) {§) {$) (?) 

Source of Cost 



Brcvsing 

Professionals 

Other $16.44 

Colleague 

Professionals 2.28 
Other 

Cited in Article 

Professionals 2.28 
Other 

Printed Index 

Prof essicHials 6 .76 
Other 1.62 

Online Search 

Professionals 1 .56 
Other 0.49 

Routed 

Profess icxials 

Other 16.44 



64,878 



$5.70 $24.64 $30.34 

16.44 



5.93 24.64 32.85 
10.06 10.06 



5.93 24.64 32.85 
10.06 10.06 



5.93 24.64 39.85 
10.06 11.68 



5.93 24.64 32.13 
10.06 10.55 



5.70 24.64 30.34 

16.44 



14,962 



41,133 



20,506 



12,559 



10,004 



aXJRCE: King Research, Inc. 
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Assimptions leading to the above costs are as follows: 



a. Browsing. It is estimated from the survey that about 50 
percent of the users vfould use anc^her library if their present 
library was not available. It was assumed that it would take 
15 minutes to get to the alternative library. Costs include 29 
percent of these articles being photocopied. It was further 
assujned that reniainder of these readings would come from 
subscriptic^s to the jcHimals and that the average readings 
would be 6 per journal. Identification is tiie same as that for 
browsing or routed cxip^. 

b. Colleague. The survey indicated that access to library 
materials would cone from (1) another library (55%) in which 
use costs would include ordering (10 minutes) , postage or 
telephone ($0.20), receipt processing (3 minutes), and leader 
costs of $13.23; (2) colleagu^author (36%) at approximately 
the same ordering costs, and lender costs at $5.15; (3) 
publisher/document delivery (9%) at ordering costs of $0.92 and 
price at $8.00 (UMI) . Identification is the same as the 
current system. 

c. Cited in Article . No identification cost and access the same 
as colleague. 

d. Printed Index . Identification adds the cost of going to a new 
library to seardi (30 minutes) for searches done by users, and 
a delay cost for searches done by intermediaries ($5.66) . All 
other costs are the same as current system. Acx:ess same as 
above. 

e. Online Search . Same identification assumptions as above. 
A::cess same as above. 

g. Routing . Same assumptiorxs as with browsing. 
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TSE VALUE OP TBE mHOm. WESCS SOFTNARE GBMTEK 
AM) THE SADIAriGN SBIELDIIC XNTCSMKnON CQITER 



1.1 BacKground 

In 1982, a study was performed for the U.S. D^rtment of ihergy 
Technical Information Center (now Office of Scientific and Technical 
Infonnation) to investigate the value of infomation generated from DOE S&D 
funding and the contribution that the Qiergy Data Base an6 its derivative 
i^rvices and products make to the value of this information. !Ihe results 
of the study demonstrated that energy information haB substantial value 
when considered from the stan^joints of amount of use, purposes of use, and 
value measured by sihat users are willing to pay for the information and the 
savings in labor, equipment, etc. derived from use of the information. 
Furthermore, energy information services and products derived from the 
Energy Data Base clearly contribute greatly to the value of energy 
information oy makiiig it more accessible and, therefore, more usable. 

It was felt that the methods and models enployed in the previous 
studies ought to be refined and an additional investigation made to 
determine what contribution intermediary information transfer organizations 
such as libraries and information analysis centers make to the vali^ of 
information. Also, other factors such as timeliness of primary publishing 
and distribution and comprehensiveness of the Energy Data Base were 
considered to be iiiix>rtant as well. Ohis further investigation was made 
under a grant from the National Science Foundation with partial 
contribution by the Department of Energy, Office of Scientific and 
Technical Information. 1!his report provides the result of the study of the 
value of information analysis centers. 

Volume 1 gives an analysis of the value of the services provided by 
libraries. The library study estimated the value of library services (in 
terms of willingness to pay and savings). It also provided the 
relationships of the use of library services and factors such as 
performance attributes (i,e., quality and timeliness of services), distance 
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to the library r awareness o£ services and econocdcs of library use. Volune 
3 provides an analysis of the value of the timeliness and cooprdiensiveness 
of the Energy Data Base. 

1.2 Introduction 

In looking at information organizations in the value added 
information chain for energy informaticn, one groi^ studied is involved not 
so much in the transfer of published information but of cocputer software 
and data. The two organizations studied that fall into this category are 
the National Qiergy Software Oenter (MESO and the Radiation Shielding 
Information Center (KSIC). Because of the differences in the form of 
infonnation dealt with, the flow of this information from author to users, 
and the uses made of software, these two organizations are considered 
separately. Figure 1 depicts the activities involved in the develc^ment 
and use of conputer programs (also called software or code packages) with 
the Involvement of a software center. 'The initial box indicates that the 
program is first developed by an individual organization to meet a 
particular need. Docunentation is developed. Vftien the program has been 
written and tested, it is put to use on a cxie time or, more frequently, a 
recurring basis for that purpose. It may clLso be submitted to a software 
center. It then goes through some level of review and testing and is 
announced in catalogs, indexes, newsletters, and other secondary products 
and services. Other organizations with a need for the same or a similar 
program can obtain the program and docunentation from the software c^ter, 
modify it as needed, install it on their particular coc^Juter, and then 
^ply it, thus shortcutting what can be a very extensive develc^>ment 
activity. Critical software center activities in this cycle are the 
acquisiticsi of programs appropriate for sharing, review and testing of both 
software and its documentation, development of secondary announcement 
products, and support to secondary users of the software. A significant 
difference in the software information chain as ccnpared with published 
infomation is the dynamic nature of the program, which is likely to be 
in^jroved and modified. 
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Docmantation 
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I 
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I 



Selecticn of Progron 



I 



Progran Modification 
and Installation 



I 



Progran Ose or Application 



I 



Program Modification 



Figure 1. Hbe Software Information Chain 
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•a is volume addresses the roles played by the National Energy 
Software Center and the Radiation shielding Informtion Center in the 
software information chain. Sections below describe the major features of 
each of the two organizations in turn and discuss the costs associated with 
the services they provide. Infoniation gathered from users on the value of 
NESC and RSIC services is also discussed. FinaLLly, results from the two 
organizations are combined to si^jpcrt the nature of the benefits derived 
fron such services and their potential relationships to productivity, 

1.3 Ht^ National Eherov Software Center 

The National Energy Software Center (NESC) is a Department of 
Qiergy Software Exchange and information Center; it is operated by Argonne 
tetiovil Laboratory, under the DCe Technical Information Services program. 
Software exchange and information center activities for the U.S. Nliclear 
Regulatory Coomission are carried out with si^jport from NRC. 

Ihe major objectives of the NESC are: 

• To promote the sharing of con^ter software among agency 
officer and contractors to eliminate di5>licaticn of effort and 
unnecessary expenditures. 

• To facilitate the transfer of computer applications and 
technology to the information-processing ccmnunity. 

• To arrange the exchange of software with other U.S. and foreign 
agencies and to assist in the acquisition of nongovernment 
software for EOE offices and cost-type contractors. 

NESC collects, packages, maintains and distributes a library of 
computer programs, models, systems routines, and data con5>ilations 
develc^jed by DOE, NRC, and their contractors in carrying out the a^cies' 
research and development activities. NESC checks the library packages to 
make certain they are cooplete and runs test cases to make certain they 
operate as described. 

NESC also prepares and distributes (1) abstracts describing the 
softv«ire packages in the NESC collection and (2) stsiwaries of software not 
included in the collection but often available on an "as is" basis. 
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!CSC Will consult with users on their software needs and will help 
them obtain the software needed. Registered organizations my obtain a 
copy of any NESC software package they are eligible to receive 
(distribution of some packages is limited). NESC will also help DOE and 
its contractors obtain needed softvare from cooioercial sources, ^le NESC 
maintains an inventory of consnercial software and issues the NESC 
Newsletter, which contains infomation on software products and services of 
interest to EOE contractors. Acquisitions of cannercial software for DOE 
use are coordinated by NESC. 

ISSC also cofnnunicates with and arranges for exchanges with other 
U.S. gcvernment software centers and with foreign computer program 
libraries and is the focal point for DOE participation in the Federal 
Software Exchange. In addition, NESC distributes software packages from 
the OECD Nuclear Eiiergy ^ency Data Bank to U.S. and Canadian requesters. 

Ihe NESC software packages are largely those developed to n^et the 
progranmatic needs of the D^rtment of Ehergy and the Nuclear Regulatory 
Oonmission. Oorputer programs and data ccirpilations arrive at NESC \itien 
submitted by agency program divisiwis in compliance with guidelines and 
contract requirements, contrikaited by authors or software development 
contractors, offered vendor exchange agreements, or solicited by NESC. 

A software package provided by NESC contains, insofar as is 
practicable, all elements required for usi^ig the software or for 
in^len«nting it in a different conputer environment. Hie packages consist 
basicadly of conputer-raedia material and printed material. Other products 
and services of NESC include coisultation with users and piiilicatiai of the 
following products. 

• The NESC Newsletter — a quarterly publication providing 
infornation on software developments by other U.S. government 
agencies and ccmnercial firms. 

• The NESC Bulletin — a bi-monthly presentation of information 
on NESC activities and recent contributions. 

• Canpilation of Program Abstracts — a catalog of programs 
2rvailable frcm NESC. 




• ICSC Notes — ncjtification of modification or correction to a 
progreun. 

Information about NESO-provided programs is also provided on the DOE/REXX)N 
online retrieval system. 

NESC services are provided prinarily to registered installations, 
including DOE laboratories, educatiaal institutions, gove ninent bodies, 
non-profit organizations and canmercial aitities. aibscripticn fees for 
different types of users provide for receipt of publications and for 
information services. Subscribers are entitled, without further charge, to 
obtain two software packages per year and additional packages are provided 
under a primary schedule. In 1983, there were about 200 registered 
subscribers at NESC. 

Infornation on the users of the products and services of the 
National Qiergy Software Center was derived fron a small survey conducted 
in the Spring of 1984. Ihe sample of the survey vras a rauidom selection of 
25 users from fiscal year 1983; 15 of whom responded to our mail 
questionnaires. V?hile this sample is quite small, it does give an 
indication of the behavior, attitude and opinions of NESC users. When 
available, survey results presented below are si^jplemented with results 
from other studies and surveys. 

Respondents to the IS£C quest ioinaire were asked about a specific 
instance of their obtaining a iioftware package from the Center. To help in 
an understanding of their flow of information we asked about the ways in 
which the re^5ondent identified the software package. The most frequent 
»3urce of information was a colleague or coworker (63%) and the other 
source mentioned was NESC publications (37%) . 

Since a najor aspect of value derived from use of the software 
center is the use of that software, respondents were asked if they had 
installed the software they received at their facility. Seventy-three 
percent indicated that they had, and another 20 percent were in the process 
of doing so. Hius, only seven percent did not plan on installing the 
software. About half the re^xjndents modified the software received to 
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sooae extent, generally because of sooe hardware-specific requiranents. cn 
the average, just under 80 hours were ^sent by those individuals vho did 
nake some modifications. Another 70 hours, on the average, was ^)ent 
installing the software packages at the users' installations. In the 
course of this installation, about a quarter of the respondents consulted 
with NESC an average of about two times each. 

Ifcat resulted from the installation of NESC-s'^lied software at 
the user installations? While the ultijnate benefits of the specific 
program to the institution are not known, some indication of value is given 
in respondents' indications of the nunber of times that software would be 
run at that institution. Estimates given by those had installed or 
planned to install the software ranged from five to "thousands" and 
"hundreds daily", averaging out to about once per working day on an annual 
basis. 

As with users of other types of infomatic^ covered elsev^ere in 
this report, NESC users of software were asked to indicate sonething about 
the alternatives to using NEsr , ihe most frequently mentioned alternative 
was to obtain coninercial software (44%) , \^ile about a third indicated that 
they would write a new program or modify an existing one themselves. Other 
responses given indicated that software might be obtained fron another 
source (17%) or that nothing would be done. Estimates of the time required 
to generate the software obtained had it not beei available ranged from 20 
hours to many person years, averaging about 1.5 persc»i years. 

To develop an indication of the overall use of NESC services, 
survey respondents were asked about their level of use within the lant 
years of the following: the NESC Nesletter. the NESC Bnlletln . the catalog 
Qocpilation of Program Abstract-R. NESC listings on the KEOON database and 
NESC consulting services. (Respondents were not asked about NESC's 
provision of software packages since all were known users of that service.) 
Results are indicated in Table 1. 
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Table 1 

/bareness and Ose of NESC Products and Services 



Percent of Respondents Average Annual 
C Product or Service Aware of Ssim Q8S&! 



lESC Newsletter 


60% 
40% 


47% 

33% 


1.7 
2.4 


MESC BulletiJi 


Gonpilation of Program 


Abstracts 


53% 


47% 


1.8 


NBSC Listings on RBCDN Database 


20% 


7% 


.3 


Consulting Services 


20% 







* Average is confuted across all users of NESC Services. 
SOURCE: NESC User Survey, Spring 1984 

As shown, the NESC Newsletter and Conpilation of Program Abstracts 
are used by nearly half of all NESC users. Ihe NESC Bulletin is used by 
about a third of the respondents, while NESC listings on RECDN (a new 
service) , and consulting services were not widely used or known. Generally, 
most of those aware of an NEl^ product or service use it. Levels ol use of 
the services, also shewn in Table 1, were about 2 uses per user per year of 
the Bulletin . Newsletter , and Conpilation . 

Another approach to understanding the value of NESC services to its 
users is reflected in their rankings of the respective services. Oonposite 
rankings by the users surveyed are shown in Table 2, and indicate an 
assignment of the highest ranking to NESC's primary service, the provision 
of software packages. Next in ranking are the NESC pi±)lications, beginning 
with the Conpilation of Prog ram Abstracts and going on to the KESC 
lifiKfiietijeilr Conputer S oftware Sunroaries. and the NESC Bulletin As 
anticipated from the levels of awareness and use, NESC listings on REODN 
and consulting services were given the lowest rankings. 



ERIC 



8 

" so 



Table 2 

User Rankings of NESC Prcx3ucts and Services 



Product or Service 



Oonposite User Ranking 



Provision of software packages 



1 



yil^tion of Program Abstracts 



2 



tESC Newsletter 



3 



:er Software Sunmaries 



4 



NESC Bulletin 



5 



NESC listings on BECOHH database 



6 



Owisulting services 



7 



SOURCE: NESC User Survey, Spring 1984. 

Infontaticn was also sought on the demogra^iic characteristics of 
NESC users. The groi;p was highly educated, with 47 percent holding a 
Doctoral degree as their highest degree and 40 percent holding a Master's 
degree. Sixty percent speit the greatest proportion of their time in 
research and development and the remaining forty percent in iranagement 
activities. A third of the group have salaries over $55,000, and the 
median salary is $50,000. Nearly three quarters of the groi^ were funded, 
at least in part, by the Federal government, with about a third funded by 
EOE. Other funding agencies mentioned were the Department of Defense, the 
National Aeronautics and Space Administration, the National Science 
Foundation, and the D^rtment of Interior. 

In addition to looking at user relations to NESC as a software 
center, this study also addresses the volume of actitivity and associated 
costs. NESC statistics reflect the activities performed and the levels of 
iise of the various products and services. Table 3 sunrarizes the NESC 
level of activity in fiscal year 1983. In the program acquisition, testing 
and processing area, 160 contributions were received, 148 screened, and 175 
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Table 3 

Ntitional Ehergy Software Center Fyl983 Activity 



355 REQUESTS FOR SOFTWARE OR AmBORIZftriOKS FOR USE FILLED 
160 SOTTWARE OMJTRIBUnONS RECEIVED; 9 FROM "fflE NEA DRTA BAMC 

148 CDNTRIBUriONS PAC3JAGED WTIB 65 OF THESE TESTED BY l. lECtmON OF 
SAMPLE PROBU»S 

97 ODNTRIBOTIONS PROCESSED FOR RELEASE ON AN "AS IS" BASIS 

206 REGISTERED OftGANIZATIONS AS OF SEPTEMBER 30 

192 ORGANIZATIONS R£);^IEErr£D MX) RECEIVED IQSC SOE^n^ARE 

94 NESC NOTES ISSUED DESCRIBING CORRECTIC»IS, IMPLEMENTATION 
INFORMATION, OR TRANSMITTAL TAPES, OR ANNOUNCING THE 
AVAILABILITy OF REPLACEMENT PROGRAMS OR AEOITIONAL MATERIAL 

3 NESC BULLETINS WRITTEN AND DISTRIBUTED 

2 SOFTWARE NES^SLEPTERS WRITTEN AND DISTRIBUTED 



released on a regular or "as is" basis, aitries were prepared for the 
Cbryilation of Program Abstrac±s and Coqputer S oftware Sumiarles. Three 
Riilletins were written and distributed, as were two Newsletters. Tliere 
were 206 registered organizations who received publications and software; 
the nurfcer of filled requests for software or authorizations for use was 
355. 
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•n^e cotts of NESC activities in fiscal year 1983 were about $820 
thousand. In order to address the question of the costs of different 
products and services, the Center's expenses were divided into five broad 
categories: program aajuisitions and processing, the Bulletin and 
NewsXgbteCf the Qpupilation of Program Abstracts and Qonputer Softwarp 
Sumoagieaf provision of programs, and other. Included in the other 
category are such in^jortant activities as involvement in the standards 
process, but for our purposes these were groi?)ed with adninistre.tive costs 
and ccr.sidered as an overhead. 

Develcping a rough breakdown of NFSC's costs in the different 
activity categories, total eatpenditures in 1983 can be seen as falling into 
the following oo.ipcnents: 

Program acquisition and processing $ 510,000 

Buj.letin and Newsletter $ 50,000 

CPA and CSS $ 80,000 

Provision of programs $ 180,000 



$ 820,000 

1.4 The Radiaticxi Shield ing Information Center 

Ihe P diation Shielding Information Center (RSIC) was established 
in 1962 at Oak Ridge Naticml Laboratory (ORNL) to serve the Uhited States 
shielding cannunity. It is sponsored by the U.S. Department of Ehergy, the 
D.S. Defense Nuclear Agency, and the U.S. Nuclear Regulatory Carmission. 
The centej: serves on an international basis and consists of scientific 
personnel who ^llect, organize, evaluate- and disseminate radiation 
protection »nd shielding information related to radiation from reactors, 
weapons, and acceleratorr; and to radiation occurring in ^ce. 
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The Center; 



• Examines and analyzes radiation protection^ radiation transport 
and shielding information obtained through surveys of pertinent 
books, journals and reports. 

• Examines digital computer codes written for radiation transport 
and shielding calculations. 

• Examines evaluated and processed cross-section data for use in 
radiation protection, radiation transport, and shielding 
research. 

• Maintains an information retrieval system which contains 
abstracted and indexed shielding informatic«i selected by its 
analysts. If an adequate abstract is not available with the 
document, it is prepared by the Ceiter. 

• Maintair;. an archival microfiche file containing copies of the 
analyze© literature. 

• Reviews specific areas of shielding to determine the state of 
the art. 

• Participates in CSEWG activities, gives leadership to shielding 
and conputing standards activities, and organizes seminar- 
workshops as needed. 



RSIC is both a software center and a more generalized information 
analysis center. As a software ceiter, it obtains and provides code or 
software packages within its subject scx)pe and also data packages. CJode and 
data packages received are extensively reviewed and evciluated. 



In addition to the provision of code and data packages, RSIC 
provides a number of related products and services, including cc»isultation 
with users and the following publicaticxis: 



• the RSIC Newsletter - a monthly information publication 
discussing new codes and data and other infomBtion of interest 

• RSIC Coc puter Code and Data C ollections — a catalog of code 
and data packages 

• other RSIC reports — various technical and analytical reports 
produced by RSIC. 



12 

75 



84 



Descriptions of RSIC packages are also available on the DG^/KE)GDK online 
retrieval system. 

Dsers of BSIC cane from all actors of the radiation shielding and 
related communities. In 1983, users represented 667 different 
installations from the U.S. and from over 50 other cotsitries around the 
world. Hiere are no subscription fees eiesociated with RSIC use. 

As with the National Energy Software Center, information about the 
use of KSIC was obtained fnxn a small survey of users. In this case, the 
sanple was selected frcin the installations with 20 or more requests in 
fiscal year 1983. All the 25 individual users surveyed were recipients of 
the RSIC Newsletter . Fran RSIC records the requests made by these 
individuals in 1983 were identified, and then each person asked about 
specific instances of use in one or more of the following categories: 
acquisition of code or data package, consultir^ or discussions with RSIC 
staff, and acquisition of a dociment. Respondents were also asked general 
questions about their use of and attitude toward RSIC products and services 
and about their deoxsgraphic characteristics. Bie survey was cMiducted by 
mil in the ^ring of 1984 1 the overall re^xxise was 14 users. 

In asking RSIC users aU}ut code and data packages they acquired, 
questioning started with the ways in \^ich the user identified or found out 
about the package. TSiere was more reliance on printed products by RSIC 
users than by litSC users, with two-thirds of the RSIC users reporting 
having identified the package through an RSIC publication. Other methods 
noted were contact with RSIC staff and a colleague or co-worker, both 
accounting for 17 percent of the uses each. 

As with NESC, nearly all (83 percent) of the recipients of code or 
data packages had installed or were in the process of installing the 
package at their facility. Fewer had made modification'?, with about a 
quarter indicating some degree of change nade to the code or data package. 
The level of effort involved in making modifications was about 80 hours per 
package on the average. 
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About half, or 56 percent, of the users of packages consulted with 
RSIC staff in the process of instiilling the pack^e. The average number of 
caonsultations was three. RSIC users sges± scinewhAt less time than IIlSC 
users in installing packages, with the average being about 40 hours per 
package. 

J^ain, one elen«nt of the value of a code or data package to the 
user is the nmtoer of times it will be run at their facility. Re^xxises 
given by RSIC users ranged widely, from five times to daily. On the 
average, about 120 uses per package were anticipated. 

Had the package acquired not been available from RSIC, users 
indicated, the most likely alternative chosen would have been to write or 
generate a new code or data package or modify an existing one. Ihis c^^tion 
was indicated by 56 percent of the users. Other alternatives noted were 
that packages inight be obtained frcm another source (22%) or tl^t nothing 
would be done (22%) . As with NESC, the nimber of hours estimated for 
generating a package were it not available ranged widely, but the average 
was about 1.5 person years. 

RSIC users were asked if they would be willing to pey for the 
package they received, and about two-thirds indicated that they would. The 
response confirms the value perceived ty users. 

In addition to their acquisition of code and data packages, RSIC 
users were also asked about specific instances of use of RSIC for 
discussions or consultations and for publications. Insufficient data was 
obtained ai publications to analyze, but half of the respondents provided 
informaticfi cn discussions they had with RSIC staff. These discussions 
ranged across a wide variety of tc^ics and v;ere well-appreciated; jII the 
respondents indicated that they had obtained the information they sought. 
Geieral categories of purposes for the discussions indicated by respondents 
included background information (57% of respondents)* information on 
methodology (29%) , conpariscn of alternative code or data packages (57%) , 
characteristics of particular code or data packages (71%), and entering a 
code or data package into RSIC (43%) . 
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Wiich of the other RSIC products and services are known, and used, 
and v^t v?ere the levels of use? Biese questions were eatplored with survey 
respondents, and the results are shown in Table 4. As with lESC, the most 
recognized and used products are the Center's newsletter and catadog. For 
both of these, all users reported awareness and 93 percent reported use, 
with average annual uses of 27 for the SSIC Newsletter and five for the 
BSIC COTPUter Qode and E&ta Collect ions. Users also reported an average of 
4.6 uses of SSIC reports and 3.0 uses of SSIC's consulting services. Ciily 
a smll percentage of users had yet nade use of the VBCm database to 
access HSIC listings. 

Table 4 

Awareness and Use of RSIC Products and Services 



Percent of Respondents A- etage Annual 
RSIC Product or Service A«aifi_Qf I^icS Uses* 



BSIC Newslettei '00% 93% 27 

RSIC R^rts 71% 57% 4.6 
BSa Cccyuter Code and 

Data Collections 100% 93% 5.0 

RSIC Listings on mm Database 57% 7% — 

Cwisulting Services 86% 64% 3.0 

* Average is coiputed ucr )ss all users of RSIC Services. 
SOURCE: RSIC User Survey, Spring 1984. 
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•Po provide another perspective, survey respondents were zisked to 
rank all of the RSIC products &nd services on the basis of their 
usefulness. As shewn in Table 5, the highest ranking was given to the 
provision of code packages, followed by package docimentaticsi and the 
provision of data packages. Ttie newsletter and catalog publications were 
ranked next in usefulness, followed by RSiC consulting services. Lowest an. 
the rzinked list were RSIC reports* abstracts, publications, and the 
relatively new REODN listings. It is interesting to note that, with cxie 
exception, the RSIC user rankings closely mirror those of NESC users — 
first, the packages themselves, then the newsletter amd catalog, and 
finally other publications. Tlie exception is consulting services, which 
RSIC users rank more highly than do NESC users. Results of ranking by RSIC 
users are similar to those obtained in a much larger survey of the groi^ 
conducted in 1977 (3). 

T&ble 5 

User Rankings of RSIC Products and Services 



Product or Service Ccii|>osite User Ranking 

Provision of code packages 1 

Docimentation of packages 2 

Provision of data packages 3 

asiC Wewaletter 4 

RSIC QfiTpu ter Qode and Qata CollectiCTig 5 

Cbnsulting Services 6 

RSIC reports 7 

RbIC abstract publications 8 

RSIC listings on RBCDN database 9 



SOURCX: RSIC User Survey Spring 1984 
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Respondents to the RSIC questionnaire were 2i£ked to indicate basic 
defnogr£^>hic characteristics, allowing us to develc^ a user profile. Most 
respondents, 71 percent, were involved in research and developnent with tJ-ie 
reroainder spending the largest prc^rtion of their tiine in management or 
cjperations (14% each). Half the respondents had Master's degrees; 36 
percent had Doctorates, and the remaining 14 percent had Bachelor's degrees 
only. A wide salary range was reported with a median salary of $35,000. 
Users cone jx>th from projects funded by the Federal goverment (57%) and 
fron other areas; the Departjment of ESiergy was the priuary agency when 
Federal funding was Indicated. 

As with NESC, the study of PSIC also involved the consideration of 
the volume or activity and aissociated costs. TSie level of RSIC activity in 
fiscal year 1983 is reflec^-ed in Table 6. As indicated, 172 packages in 
total were contributed including new programs and data libraries, new 
hardware versions, and i5)dates and error corrections. Seventy-one packages 
were processed for announcement and distribution. The Newsletter was 
developed and distributed, and in the use area, a total of 3,213 letters 
and calls requesting 9,857 products and services were received. Among the 
products and services provided were 1,050 oode or data padtages, 150 
additional docun^ts, and 8,570 responses to inquiries. 

The costs of RSIC activities in fiscal year 1983 were about 
$900,000. In order to look at the product and service costs specifically, 
total expenditures were subdivided into: package acquisition and 
processing, piislicaticns and provision of packages, consultation, research 
and developnsent, and adninistrative and other activities. Considering the 
expense of administrative and other activities as overhead, the rough 
breakdown of RSIC costs derived was as follows: 



Package acquisition and processing 

Publications and provision of prograns 

Consultation 

Researc' ' Develo{inent 



$ 900,000 



$ 50^,000 
9 170,000 
$ 110,000 
$ 120,000 
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•Table 6 

RSIC Activity Statistics (Fiscal Year 1983) 



TECHNDLOGY CDNTRXBOTED: 

• 89 New coEputer programs and data libraries 

• 18 New hardware versions to extend existing co<3e or data packages 

• 22 U|x3at cs for error corrections discovered in using existing 

cod^tfeta packages 

• 43 Updates to existing code/data packages 

TEOiNDLCXJY PRXESSED: 

• 26 New code packages 

• 5 New data packages 

• 7 Updates to include conversions of RSIC co^s packages to run on 

other harti^re 

• 9 Newly frozen coirputer program versions including iirprovements 

made over that originally packaged 

• 5 Updates to data packages 

• 19 Updates to code packages 

• 71 Total packages processed 

PUBLICXriOie: 

• ijunthly Newsletter to a peak of 1664 people 
DISSEMINflTION: 

• 3213 Letters and calls requesting 9,857 products and services 

• 1050 Cod^data packages provided 

• 150 Additional documents provided 

• 8570 Responses to inquiries for information 



ERIC 



l-j 
81 

90 



1.5 



The Vhlue of Software Center Pr oduc±fi ant^ ^r-^^r? ^ 



As has been Indicated, both !ESC and IBIC provide a considerable 
range of prodix±s anc*. services related to software and data packages. 
Their publications loth provide inforinaticn about the packages and serve 
additional purposes such as to inform subscribers about commercial 
software. C3r«isultation services provided, especially by RSIC, are not only 
associated with the provision of packages but may also be concerned with 
data, techniques, and programning in general. Both SSIC and NESC, in 
addition to the acquisition and distribution of code packages, are also 
involved in such professional activities as standards develc^inent , 

The two organizations, lESC and RSIC, have a general mission in 
coiiiDon but are otherwise fairly different in their operations. Subject 
scope, as indicated earlier, is different, with NESC having a broader 
scope. The level of processing also varies, with NESC processing some 
contributions for release on an "as is" basis and RSIC, in general, doing 
more elaborate testing. Dq)ending primarily on their user ca..Tiunity, the 
two organizations have also developed different types of supporting 
services. 

nhe major concern in this section of the report is with NESC and 
RSIC as providers of software packages, and with the value associated with 
that service. In that regard, we view the two groups together and 
consider, in turn, their costs of operati and volune of service provided 
as indicatiwiF of the value derived from these services. 

In fiscal year 1983, the total expenditures of NESC and RSIC were 
$1.7 million. Of these costs, $1.0 million was spent on package 
acquisition and processing and $700 thousand on distribution and related 
services. Between the two organizations, with their different levels of 
processing, about 250 packages were processed and 1400 packages were 
provided to users. Oie set of unit costs that can be derived frcci these 
nunt)erp is $6, COO per package input and $500 per package provision. More 
properly, the total cost of providing one software package, on the average 
and including acquisitidi and related service costs, is about $1,200. 
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OSiere are, of cx>urse, other costs involved in tl»e provision and use 
of energy software. We do not know, for exanple, what the original costs 
of develc^ing the software in NESC and RSIC were. In terms used elsev^ere 
in Voluae 1, these might be considered authorship costs. Other costs in 
the dissemination chain would include the cost of submitting the software 
to a Center, the costs of identify iiig the C3enter as a source, and the costs 
of modifying and installing a package once it is received. Based on user 
estimates obtained in the user surveys, modification and installation 
activities for all packages received from the two Centers averaged about 75 
hours or $2,700. Adding this to NESC and RSIC costs for their services, we 
estimate a total cost of $3,900 per installation of an energy software 
package. The cost to the user on the average is the $2,700 labor cost 
incurrred plus an approxinate $700 share of the NESC subscription fee paid. 
This yields a total effective price of $3,400 per use. T^t is, the users 
indicate thsy are willing to pay $3,400 per installed package (1,160) or 
$39 million in toteil. 

To address the valui of the NESC and RSIC-provided software 
services, we can first review the costs of obtaining the same packages from 
other sources. The most likely alt .native source indicated by respondents 
%*as in-house developmeit of the pr'i^gram; this was estimated on the average 
to require about 1.5 person years of effort or nearly $68,000 in labor. 
This suggests that, on the averaQe, an investment of $3,900 in using one of 
the software centers leads to a savings of $64,000. i^lied across the 
1160 packages provided and installed, this suggests a total savings of 
about $74 million. 

Hie value of the services is caaputed by determining wiiat would 
happen if the services were not available. Hie cu rent total cost of 
acquiring the software packages is $4.8 million (l,4lj / $3,400). If it 
costs $68,100 to prepare these packages in-house, the users could only 
prepare 70 software packages with the $4.8 million now being i^?ent. "flius, 
1,09C packages would be lost (1,160 minus 70). The value in terms of 
willingne£>s to pay of these lost software packages is $3.7 million. 1!he 
value in terms of savings to users of these services is $74 million. The 
investment in these services seems clearly worthwhile. 
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Another indicaticn of value would reflect the value of the use made 
of the software package. While this was not explained in the user survey, 
it is known that the software packages provided lay IGSC and SSIC are used, 
on the averager about 150 times. "Diis suggests^ at least, that there is 
considerable value derived from them. 
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SECTION 1 

XNnoDucnoN 

1.1 Background 

In 1982 a stu^ was perfonned for the U.S. Departinent of Qiergy 
Technical Information Center {now Office of Scientific and Technical 
Infomiaticn) to investigate the vzLLue of information generated from DOE R&D 
funding and the contribution that the Biergy Data Base and its derivative 
services and products make to the value of this information. Ihe results 
of the study demonstrated that energy infonration has substantial value 
when considered from, the standpoints of amount of use, purposes of use, and 
value measured by v^t users are willing to j^y for the information and the 
savings in labor, eguiproent, etc. derived from use of the information. 
Furthermore, energy informaticxi services aiid products derived from the 
Eaiergy Data Base clearly contribute greatly to the value of energy 
information by making it more accessible and, therefore, more usable. 

It was felt that the methods and mo^ls enployed in the previous 
studies ought to be refined and an additional investigation made to 
determine what contribution intermediary information transfer organizations 
such as libraries and information analysis canters make to the value of 
information. Also, other factors such as timeliness of primary publishing 
and distributiai and caiprshensiveness of i-.be Energy Data Base were 
considered to be important as well. This further investigation was made 
under a grant from the Natic»ial Science Foun^ition with partial contribu- 
tion by the Department of Ehergy, Office of Scientific and Technical 
Information. This volune provides the result of the study of the effect 
that timeliness and conprdiensiveness have en tlie value of information. 

Generally, we looked at timeliness ficm the standpoint of the 
duration of time from information generation to publication of articles and 
technical reports and document delivery time (i.e., time frcxn determination 
of information need to its delivery). Timeliness of bibliographic 
publication and speed of response in online and manual searching performed 
by reference librarians was also addressed. In effect, it was determined 
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that delays in any of these processes can result in readings or uses being 
lost and sucfc losses in turn reducing the vzQue of the informtion; ^^re 
vzLlue is measured in terms of 'vibat end-users are willing to for the 
information and the dollar savii^s that result fron having road and used 
the information. Ihis analysis is reported in Section 2. 

Hie question concealing the value of non-US materials was also 
investigated. This is done by determining the extent of use of non-US 
materials, determining substitute means of getting these materials, 
establishing the readings that would be lost under a given budget, and 
estimating the vcilues (i.e., willii^ess to pay and savings) that these 
lost readings represent. Analysis is performed from surveys of the 
scientists and engineers from nine fields of science and special surveys of 
DOE funded scientists and aigineers. Results are presented in Section 3. 

A special kind of analysis was performed on a sanple of searches 
v^ich were conducted for DOE funded scientists and engineers. Oie purpose 
of the analysis was to get seme indication of use of ncxi-lB materials from 
the search outputs and to evaluate the use of "cross-cutting" research by 
D0E-fi2ided scientists and engineers. This was done by conparing the 
subject categories of online searches that were performed by DOE fimded 
researchers against the subject categorization of the research project for 
which the search was being performed. We also determined the relevance of 
search response to the research project as well as to general interest of 
the researcher. Results of this investigation are discussed in Section 4. 

In addition to this volume there are two other volumes. Volume 1 
is a report on a study of "The Value of Libraries as Intenaediary 
Information Services". This study attempted to measure the value of 
library services (in terms of willingness to pay and savings) . It also 
provided evidence as to factors that contribute to the extent of use of 
library services. Such factors include library performance attributes 
(i.e., quality and timeliness), distance to the library, awareness of 
services and economics of library use. Finally, a study of Information 
Analysis Centers eis intermediary information services was daie. Its resort 
is Voltsne 2, "T5ie Value of the National Energy Software Center and the 
Radiation Shielding Information Cfenter." 
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1.2 Stiu^ Methods 



Ptor this study, data collected f rc3in three surveys were use»^: 



(1) A survey of the population of scientists and engineers from 
nine fields of science was conducted as part of a study being 
perfonned for the National Science Foundation (Statistical 
Indicators of Scientific and Technical Ccranunication) . This 
survey involves estiitating the extent of authorship of journal 
articlesf journal reading, journal subscriptions, technical 
report reading, library use, nimeric data base searching and 
bibliographic data base searching. Demographic information 
identifies scientists and engineers v^o are funded by DOE. 

(2) A survey of the professionals esnployed at Rocky Flats, Rockwell 
and ORNL was also conducted. This general informatitxi usage 
survey obtained infomation about general reading, library use, 
awar&iess of services and satisfaction with services. It is 
noted that this survey included managers, administrators and 
operational professionals as well as scientists and engineers 
engaged in KuD. 

/ 

(3) In addition, four sun'eys were conducted at the three library 
locations to determine detailed infOinnation about recent uses 
of specific library inaterials and services: online searches, 
manual reference searches, journal articles and technical 
reports. Data collected included sources of services, value of 
readint,, purposes of use, other value related information and, 
for the two types of searches, conjoint measurement data. 



Hie survey of scieri1r "^ts and t-igineers actually was done in two 
parts. The first part obtaL.ssKS information about jouriHl article author- 
ship, readint, (i.e., amount, means of identification tind access, purpose of 
reading and value derived from reading), and journal subscriptions, l^e 
second part concentrated on technical report reading, library use, and 
online searching from niffneric and bibliographic databases. Saiiples for both 
parts were chosen randomly from lists of scientists and engineers provided 
by professional societies. Ihe first part involved a mail-out of 1,958 
questionnaires, of which 1,252 re^nded {64 percent re^x)nse rate). Of 
these 588 of the scientists and engineers indicated they were not involved, 
thus yielding 664 questionnaires that were analyz«>d. Olie second part had a 
mail-out of 1,017 questionnaires of which 467 responses were received and 
195 were analyzed. The second survey involved EOE funded professionals 
(scientists and engineers and others) who work at Itocky Flats, Rockwell 
Energy Systems Group and Oak Ridge National Laboratories. 
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Ihe profsssianals were randcoily chosen from lists provided by th^ organiza- 
tions maitioned above. A total of 200 questionnaires were distributed and 
137 received {69 percent response rate) . To achieve this level of response 
the questionnaires were distributed by the libraries (or KRI) to a random 
sanple of professionals who work at the three locations. Soce data were 
collected by telephone follov-up in order to achieve desired response 
rates, ^e crapleted quest ionr^ ires were returned to KRI by the respon- 
dents by a self-addressed envelope in order to ensure ccmf identiality . 

•Hie last set of surveys involved a total of 143 questionnaires that 
were distributed to recent library service users (41 article readers, 35 
technical report readers, 38 online search users, and 29 iianual reference 
search users). The follcwLig nunt)er of questionnaires were returned: 30 
article readers, 22 technical report readers, 30 online search users, and 9 
nanual reference search users. The overall response rate was 91 percent. 
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SECtltDN 2 

TOE MLAnONSHiP OF TIMELINESS TO VALUE OP INTORMMIC*! 



2.1 Background 

In this section we address the question of how inportant tiitieliness 
is to the value of information . Biere are several a^sects to timeliness 
including (1) the tiinelinesE of publication; that is, the duration of tijue 
between generation of infonnation to the date it is published; (2) the 
timeliness of docunent delivery, that is, the duration oi time between the 
date a need for information arises and the date an article or technical 
report is delivered; (3) timeliness with \Aich the dociioents are announced 
in bibliograj^ic services, and (4) timeliness of delivery of bibliograj:hic 
search results. Presumably, delays in any of these activities can result 
in needed information not being available and therefore, not used, which 
reduces the value of the infonnaticn. Thus, reducing the duration of the 
activities has the potential for increasing the value of the infonnation. 
The thrust of this investigation is to determine the extent to which 
information is needed and the value that is lost because it is not 
available when needed. 

Information, whether found in books, journal articles or technical 
reports, is read and used for many purposes including research, education 
of oneself, eAx:ation of others, and manp.gement. Sometimes reading and use 
of the information can take place any time during a wide span of time 
without any appreciable effect on its value, cn the other hand, soietimes 
information must be read and used consequent to other events or it loses 
much of its value. For exauple, data that can replace an e3q)eriment must 
be known prior to the experiment for it to be of value. Knowledge of an 
environmental regulation must be known prior to an e^riment involving 
toxic substances or construction of a nuclear site. Finally, there are 
instances in v^ich informatics has to be read and used within a very short 
period of time or it totally loses its value. For exanple, information 
used in i»reparing a proposal muf5t be used prior to the proposal's submittal 
date or :t loses its value, or at least the value derived for that purpose. 
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Evidence fran previous studies suggests that much of the reading of 
journal articles is for self -education, keeping aware of %#hat is happening 
in the field and satisfying one's curiosity. In these instances, 
timeliness is not critical, Hcwever, there are times v<hen infornation from 
journal articles is needed fast (i.e., within a day) . IJhless the scientist 
has the article or can get it from a co-worker or the library, the 
opportunity to read the article in time may be lost. Currcsitly, there is 
little chance for getting a copy quickly by interlibrary loan or docisnent 
delivery service, although services such as overnight delivery and the UMI 
Article Clearinghouse are inproving the situation. There seems to be less 
casual reading of technical reports than journal articles because they are 
less readily available (i.e., journal issues are widely distributed) and 
the effort is greater in getting and reading the technical reports. For 
this reason, timeliness may be a greater factor, particularly in timeliness 
of document delivery. Finally, timeliness of identification through 
bibliographic products and services can also be an important factor in the 
use and value of infomatiwi found in both journal articles and technical 
reports. 

2.2 Effect of Publishing Delay on Value of Infontation 

There are three factors that one must consider in determining the 
effect of publishing delays cti value of information. The first factor is 
the delay time between research discovery or project results and the tiit^ 
in which it is published in technical reports, journals an^or books. The 
second factor deals with the nianber of readings that would have otherwise 
occurred during the period of time in which the delay took place. The 
third factor is the value ol- these readings in terms of users' willingness 
to pay for the information and by the savings that are incurred by users as 
a result of having read the publications. In this analycis we concentrate 
primarily on readings of technical reports and journal articles. The 
reason for not including books is that they are read relatively 
infrequently and the information found in them is rarely as current as that 
found in technical reports and journal articles. 
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An indication of what might read if pii>lication8 are pro</ided 
earlier is the distribution of reading over time as shown in Table 1 below. 



Table 1 

Age of Technical Reports and Journal Articles Read 
by Scientists «id Bigineers - 1984 

(%) 



Year 
Published 



Technical Report 
Readings* 



Ooumal Article 
Readings** 



1984 

1983 

1982 
1980-81 
1975-79 
1970-74 
< 1970 



51% 
23 

9 

9 

3 

3 

2 



71% 
12 

4 

4 

5 
2 
2 



Average Age (Years) 



1.4 



1.3 



*** 



The survey was taken in May 1984 so responses in 1984 reflect only 
four inonths. 



Source; King Research, Inc. - Survey of Scientists amd Engineers (n*=195, 
n**«€64) 



The survey was conducted in May, 1984 so that readings of publications 
published in 1984 represent current reading; in fact, much of the journal 
article readings occur as a result of browsing current journal issues. Bie 
average age of technical r^rts and journal articles read is nearly the 
saine. However, the shape of the distribution of age is scme^^t different 
in that 71 percent of the articles read are less than four monfis old, 
coc^red to 51 percent for technical reports. It is noted that many 
techrdcal r^rts read are internal reports {i.e., reports prepared within 
the scientist's organization). Scientists and engineers funded by DOE 
indicated that 28 percent of their technical report readings came from 
materials published in the past four months. T^us, it appears that DOE 
funded scientists and engineers have a better means of getting curroit 
technical report publications than scientists and engineers generally. 
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It is not known exactly what the publication delays are for 
technical reports. However, evidence from the National Technical 
Information Service and elsewhere suggests that it takes about three to 
four months for typical technical reports prepared uider government 
contract to be formally pdalished by the sponsoring agency. Belcw we give 
some indication of the amount of reading, usefulness and value of reading 
that is lost Airing this lapse of time. As stated earlier, it is felt that 
most technical report reading is in re^xsnse to ^jecific needs, reflecting 
research (or other events) that involves information needs occurring over 
relatively brief, but ^)ecific periods of time. That is, there is probably 
far less serendipitous finding of valu-^ble information in technical report 
reading than in journal article readLig. O^iis conjecture is reflected in 
the fact that only about cne-fourth of the journal article readings yield 
savings as a consequence of the readings, whereas across all technical 
report readings, this prc^rtion is found to be 41 percait (68% for DOE 
funded scientists and €5igineers) . 

Clearly, some value is lost for publishing delays for both 
teclmical reports and journal articles. I^Qse values are suijinarized in 
^I^le 2 below. 



Table 2 



Value Lost Due to Publishing Delays for 
Technical Reports and Journal Articles - 1984 



Value Lost 



Type of Bteterial 



Willingness 
to Pay 



Savings in Labor 
and Eguipnent 



Technical Reports 
Journal Articles 



$1.5 million 
$3.1 million 



$10 million 
$33 million 



SOURCE: King Research, Inc. 
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Evei though these figures are calculated oased <xi sore rough asaumptions, 
the orders of magnitude indicate that publishing delays warrant 
investigation. Hie assmitionE and methods of calculating values lost due 
to publishing delays are discussed belcw. 

^ere are several assun|:tions that went into the estimates of value 
lost due to publication delays of technical reports. These assLBifJtions and 
their effects on lost readings are sumarized in liable 3 below. Note that 
there is a proportion of reading involved with each assutption. Hhis 
proportion is multiplied against the reraining readings to determine how 
many remain after the eissumption is made. 



Table 3 

Assiarptic^s Used to Calculate the Ntaifcer of Technical Report 
Readings Lost IXie to Publication Delay 





AssLS^ion 


Prcportion 
of Readings 
Involved 


ffcirber of 
Readings 
Left 


1. 


Some technical reports would have been useful 
if they had been published earlier 


.27 


1,782,000 


2. 


Recently read technical reports are more 
likely to be involved 


.70 


1,247,400 


3. 


Informaticxi in some of these technical 
reports is unique (i.e., readers are not 
aware of other sources of the ^jecific 
information or equally useful information 


.40 


498,960 


4. 


Reader already knew about the research that 
was reported, therefore could have gotten 
the information frcn the author (65%) 


.35 


174,636 


5. 


Publication delay is four months over a four- 
year period in v^ich information could have 
been useful 


.083 


14,495 



Scientists and engineers wer»2 asked several questions concerning 
the last technical r^rt they had read (i.e., they reported on a "critical 
incidence"). One question concerning this techni«d report was v^ether the 
information found in the report would have been useful in the course of 
their research (or teaching) if it had been published earlier. Responses 
to this question indicated that 27 percent of the read technical reports 
would have been useful, if they had been published earlier. It was esti- 
mated in 1981 that there were about 6.6 million readings of technical 
reports by IDE funded scientists and engineers. Thus, if 27 percent of 
these would have been useful, if published earlier, there would have been 
1,782,000 such readings.* Secondly, it is assumed that such lost read- 
inys from publishing delays would be more significant from recently pub- 
lished technical reports (i.e., published in 1984). About 70 percent of 
these te dinicaa reports fall into this categon^. Thus, the uses tl^t would 
have been made from recently published technical reports are 1,247,400 
readings. 

The readers were also asked whether the information obtaij"'':;.. in the 
read technical report, or equally useful information, could have been 
obtained elsewhere. It is assumed that, if the information was not unique, 
it could have been obtained elsefe^ere, evoi if the technical report was 
published earlier. About 40 percent of the technical reports (published in 
the past four mmths and which were indicated to have same use if published 
earlier) were indicated to have unique information.** Thus, the remain- 
ing readings with the above characteristics would be 498,960 readings. 
Also, in "ome instances, the readers already knew about the research 
reported in the technical reports. If they did, we assumed that the reader 
could have gone to the author to get the information even if the technical 
report had been published earlier. It was found that 35 percent of the 
technical reports had this characteristic, so that the number of readings 
that would have happened if the technical r^rt had been published earlier 



* Note the four significant places (1,782) are maintained in the estimate 
merely to help clarify the example. Clearly, such degree of accuracy 
from a survey is spurious. 

** Note that, if one does not limit the calculations to recently published 
technical reports, this proportion would be 38 percent. 
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is 174,636. Unfortunately, we did not determine when, prior to 
publication, the technical report would have been used. We assumed that it 
would have been used soitietiine between the publication date and four years 
prior to that. If these uses were evenly dis( ributed over the four years, 
only 8.3 percent of these readings would be lost due to a four-mcaith 
publishing delay (4-48). "ttius, the rtanaining 14,495 readings represent 
those that would be lost due to publishing delay, under all the assunptions 
made above. llie average value of technical report readings is $102 in 
terms of i/Aiat users are willing to pay for the infomation and $706 in 
terms of the savings that are derived as a result of having read the 
technical r^rts. 'Rwis, the total values for the 14,495 readings lost 
would be about SI. 5 million for willingness to pay value lost and $10 
million for savings value lost. 

For estijtBting value of publishing journal articles earlier, we 
mke a distinction between reading of recently published articles (i.e., 
less than four months) and the renaining readings. In 1981/82 there were 
estimated to be 7.1 million readings of journal articles by DOE funded 
scientists and engineers. 

We do not have data corresponding to the usefulness of the 
infonnaticn in the article if it had been published earlier. However, if 
we assume similar needs to that observed with technical reports, a rough 
estimate can be derived. First, it is assumed that 19 percent of the 
recently read articles (less than four maiths old) would be more useful if 
published earlier. These, and the other assunpticsis, are given in Table 4, 

If we apply the assun^Jtions above to these readings, about 19 
percent of the readings would be from recently published articles and 
useful prior to their current publication (1,341,900), Of these, 60 
percent of these readings involve unique information and information about 
research that was not known about before. TSius, about 805,140 of the 
readings would be useful and also be unique. About 40 percent of the read 
journal articles had research that was not known about before, which is the 
proportion used to compute the lost readings (322,056). Furthermore, with 
a 12.9 month publishing delay, the probability over a four-year period that 
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Table 4 



As8un|>tic3ns Used to C&lculate the Nunter of Journal Article 
Readings Lost Due to Publication Delay 





AssuR]|ition 


Proportion 
of Readings 
Involved 


Nlmber of 
Readings 
Left 


1. 


Sane journal articles would have been useful 
if they had been published earlier 


.27 


1,917,000 


2. 


Recently read journal articles are nore 
likely tv; be involved 


.70 


1,341,900 


3. 


Inforrotion in seine of these journal 
articles is unique (i.e., riders are not 

inforration or equally useful informaticxi 


.60 


805,140 


4. 


Reader already knew about the research that 
was reported, therefore could have gotten 
the information fran the author (65%) 


.40 


322,056 


5. 


Publication delay is 12.9 months over a four- 
year period in which information could have 
been useful 


.27 


57,970 



readings would be lost due to the publication delay is about 0.27. Thus, 
the total journal article readings that might be read and savings value 
derived, if published earlier, is estimated to be about 86,955. The 
willingness to pay and savings value for reading journal articles are $36 
and $385 respectively, so that the total values are about $3.1 million and 
$33 million respectively. 



'i^iere are some interesting aspects of the publishing delays and the 
characteristics of the technical reports and articles that are discussed 
below. This information might also be used to form different assunptions 
for calculating nurrber of readings lost due to publishing delays. 



As mentionec above the average delay in publishing journal articles 
is about 12.9 months "rom the time an article is subnitted for publication. 
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The delay in publishing journal articles appears to be increasing, based on 
surveys conducted in 1977 and repeated in 1984. Results of these surveys 
are given in Taole 5 below. 



Table 5 

Elapsed Time Between Article Manuscript Submissiai and Publication 

by Field of Science - 1977 and 1984 
(Months) 



Field of 


Elapsed Time Between Submission and Publication 


Science 


1977 


1984 


Physical Sciences 


8.0 months 


8.9 months 


Mathematics 


20.5 


17.0 


Cofilx±er Science 


10.6 


12.1 


Ehvironmental Science 


14.4 


17.2 


Ehgineering 


9.0 


9.9 


Life Sciences 


12.1 


12.7 


Psychology 
Social Sciences 


12.1 


15.1 


10.3 


13.1 


Other Sciaices 


5.8 


8.3 


Total 


10.3 months 


12.9 months 



Source: Kiiig Research, Inc. -- Survey of Sci^tists and Engineers (1977 
n=600,1984 n=213) 



It is noted that the publishing delay in 1984 is greater than that in 1977 
in every field of science except mathematics. 

Scientists and engineers were asked whether they knew about the 
research reported or discussed in the article they r^d prior to reading 
about it. Not surprisingly, about 40 precent of the incidences in which 
recent publications (published less than four mcxiths ago) were read 
involved research that was not new. However, up to a point, as the 
publications get older, the scientists and engineers are more likely to 
know about the research, ^en they iDegin to be less likely to know about 
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tjie research. Ihis imy be because sane scientists recently entered the 
field and were not a««are of the research or their mesoories may have Ij^sed. 
The results of knowledge of the research are given below in Ti^e 6 below. 

Table 6 

Proportion of Readings in Which Scientists and Eiigineers Knew 
About the Research K^rted in Journal Articles by 
Year of Publication - May 1984 

{%) 



Year Article Was Publi«Apd 

1984 1983 1982 1930-8 1 1975-79 107Q-74 < 1Q7Q 



Knew About Research 


40 


46 


74 


72 


S3 


58 


29 


Did Nbt Knew 


€0 


54 


26 


28 


37 


42 


72 


Observations (n) 


421 


79 


23 


29 


32 


12 


14 



SOURCE: King Research, Inc. - Survey of Scientists and Ehgineers (n=664) 



Analogous results are given for technical reports in Table 7 below. 

Table 7 

Proportion of Readings in Which Scientists and E>igineers Knew 
About the Research Reported in Technical Reports by 
Year of Publication - May 1984 

C%) 



Year Article Was Published 

198 4 1983 1982 1980-81 1975-79 1970-74 < 1970 



Knew About Research 


65 


67 


70 


50 


75 


75 


33 


Did Not Know 


35 


37 


30 


50 


25 


25 


67 


Observations (n) 


65 


30 


10 


12 


4 


4 


3 



SOURCE: King Research, Inc. - Survey of Scientists and Engineers (n=195) 
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Ihe results show that knowledge of the research r^rted seenis to 
be greater over tiine (although saitple size may be too snail to say for 
sure) . Hie higher knowledge throughout the ages of the publication for 
technical reports versus journal articles supports the notion that more 
serendipitous identification cones from journal articles. 

The scientists and engineers were also asked whether the 
information obtained from the article read (or equally useful infomation) 
could be obtained elsewhere. Most of the people viho indicated that the 
information could be obtained elsewhere (e.g., from the author) also 
indicated a knowledge of the research reported. The publication age 
pattern of reading of noi-unique infomaticn found in articles followed 
that of the knowledqe of the research in Table 6. Coi^arisons are given in 
l^le 8 below. Both patterns begiri at 40 percent in 1984, peak in 1982 and 
end at 29 percent and 20 percent, respectively. 

Table 8 

Proportior of Reading in Which Scientists and EJigineers Knew 
About the Research R^x)rted in Journal Articles and 
Felt 1'hey Could Get the Information 
Elsewhere by Year of Publication - May 1984 









Year Article Was Published 






1994 


1983 




1980-81 


1575-79 1970-74 < 1970 


Knew About Researci) 


40 


46 


74 


72 


63 


58 29 


Could Get Inform- 
ation Elsewhere 


40 


44 


53 


39 


36 


27 20 


Observat ions 


334 


59 


19 


23 


28 


31 10 



SOURCE: King Research, Inc. - Survey of Scientists and Eiigineers {n=664) 
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Results for technical reports are givai in Table 9 below 



Xgible 9 

Proportion of Reading in Which Scientists and Eirgineers Knew 
*About the Research Reported in Technical Reports and 
Pelt They Oculd Get the Inforraaticxi 
Elsewhere by Year of Publicaticn - Hay 1984 



Year Article Was Published 





1984 


1983 


1982 


198P-81 


1975-79 


1970-74 < 1970 


Knew About Research 


65 


67 


70 


50 


75 


75 33 


Oould Get Inform- 
ation Elsewhere 


45 


48 


44 


30 


25 


50 100 


Observations (n) 


55 


27 


9 


10 


4 


2 1 



SOURCE: King Research, Inc. - Survey of Scientists and Qigineers (n=195) 



2.3 Effect of Dela ys in Dis tribu ting Document Copies on Value of 
Information 

We first looked at the speed with v^ich documents were obtained 
following the need for (oi- interest in) them. Results of the speed of 
delivery of technical reports is as follows: 

Sj peed of Delivery PlPpOCtion Qf Readings (%) 



Iimediately 17 

Within one day 9 

In two days 26 

In three to five days 30 

In two weeks 9 

Over two weeks 13 



Seme of the technical reports were obtained inroediately because they were 
read from routed copies or while browsing through the shelves or from 
microform copies of the technical reports. 
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The scientists asid engineers were also asked to indicate the 
required amount of time (in tenns of greatest value) to get the technical 
reiwrts. Bere we find that about one-third of the docianwit deliveries were 
made at the required amount of tiaie, 28 percent were obtained faster than 
requiredr and 39 percent slower than required. In nearly all Listances in 
*^ich the technical r^rts were obtained after they were required, it was 
indicated that they were needed very fast (within ndnutes or hours) . TSiey 
actually got them in one or two days (about 20%) , three to five days (about 
40%) or over one week (about 40%) . It is in these instances, that some 
value is lost. Of the readings in which cane value is lost, about 60 
percent of the reports apparently were ordered by the library. Of those 
ordered by the library, (30% of all readings) abuut 43 percent were needed 
within one day. ^us, about 13 percent of all the technical leport 
readings could potentially save value i.f they could be delivered faster to 
the library. 



Another w^ to look at speed of delivery is hew much use would be 
lost if document delivery was delayed over aie week of the actual delivery. 
Here it is found that r±)Out 12 parent of the technical report readings 
would be less useful and eight percent of these would not be useful at all. 
If we apply the ei^ht percent to the figures above as an indication of 
value that would be lost, the total readings of technical reports in v^iich 
value would be lost due to late delivery would be about 16,000 readings 
(.13 X .08 X .23 X 6.6 millicsi) . Value in terms of willingness to pay of 
the lost readings would come to about $1.6 millicsi. Savings value of these 
readings would be about $11 million. Again, even recognizing the "soft" 
nature of these estimates, speedy delivery, when needed, should yield 
substantial value. The same holds true with delivery of journal article 
separates as well. 



Copies of journal articles are more accessible, because individuals 
and libraries subscribe to journals in anticipation of their use, btised on 
past experience with the journals. Generally, individuals do not subscribe 
to journals unless they frequently read thesn. Otherwise, they rely on 
libraries to provide them with copies in which to read. It is also found 
that it is less expensive to borrow (i.e., get a photocopy) article copies 
fro^ another library or order one from a document delivery service than to 
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sii>scribe to a journal if the nunber of readings of the journal falls below 
the accepted range of ten to 20 readings (depending on price) . Thus, 
libraries can provide- a useful service to individuals tiy ordering copies of 
articles they do not have. About 11 percent of the readings involve such 
orders of s^rate copies of articles. We have estijnated that 28 percent 
of these orders require getting the article cc^ies within a day. Hhus, 
about three percent of the readings involve "rush" orders. If, as above, 
eight percent of these readings would lose their value if not delivered 
rapidly, the total savings value lost would be about $6,6 million on the 
17,000 such readings. Readers indicated that they would be willing to pay 
an additional $7 per reading on the average for the 213,000 readings in 
v^ich they require "rush" deliveries, so that the total value they vwuld be 
willing to pay for these is about $2,9 million ($43 x 213,000), 

2,4 The Effect of Delays in Publishing Bibli ographic Information on the 
Value of Information 

There are a nunt^er of ways that technical reports and journal 
articles are identified. Two of these ways involve bibliographic services 
and products such as the Energy Research Abstracts (ERA) , a printed 
publication of abstracts and indexes, and FECDN, an cnline bibliographic 
database. It takes some time to produce bibliographic databases and the 
services and products fran these databases. A quest ioi arises as to the 
value of information that would be gained, if the databases and their 
derivative products would be produced and made available faster. This 
question is answered in part by the amount of reading that occurs frcm the 
bibliographic publications and search services. It also depends on the age 
of mteriais read from these services. These factors are discussed below. 

A survey of Department of Eiiergy funded scientists and ^gineers* 
showed that the majority of readings of technical reports are identified 
through distribution methods such as routed copies or accessioi lists 
(52%). Many of the technical r^xjrts are also identified via referrals 



* The initial survey reported in "Value of the Ehergy Data Base," King 
Research, Inc., March 1982. 
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from the readers' colleagues (24%), as well as by their citation in other 
technical reports or journal articles (8%) . It was estimated that this 
population of scientists and engineers also frequently identified read 
technical reports through printed indt^xes (16%) and computerized 
bibliogr^ic searches (12%) . Similar results are observed for inethods of 
identifying journal articles .by DOE funded scientists and engineers v^ere 
browsing (42%), colleagues (18%) and citations in articles (15%) account 
for most of the reading. However, seven percent of the article readings 
were identified through printed indexes and 18 percent through ccnputerized 
bibliographic searches, 

A survey across all scientists and engineers suggests that they do 
not identify technical reports as frequeitly through bibliographic services 
and products (5% printed indexes, 4% ccsiputerized searches) as do the DOE 
funded scientiots and engineers (16% and 12% respectively). These 
proportions for the entire population of scientists and engineers were six 
and three percent respectively, which confirms the evidence that the DOE 
funded scientists and engineers search bibliographic publications and 
online databases much more frequently than other scientists and engineers. 

The age of technical r^)orts and journal articles through various 
method- of identification are given in Tables 10 throi^h 13. The average 
age of items identified through printed indexes is 1.0 and 1.2 years for 
technical reports and journal articles respectively, while the average age 
of items read frcm axiputerized bibliographic searches is 2.4 and 4.3 years 
for technical reports and journal articles, l^is may suggest that printed 
indexes are used more for current awareness (or at least for searching 
recent bibliographic publicaticxis) and con^uterized bibliographic searches 
for identifying older materials. The ages of the items read from 
caiputerized searches cx^npares reasor^ly closely with the itesns identified 
through citations in technical reports, journal articles, etc. 

The results suggest that a fair amount of reading comes froB newly 
published materials identified from bibliographic services and publi- 
cations. In fact, with DOE-funded scientists and engineers, this number is 
estimated to be about 350,000 readings of technical reports and 550,000 
journal articles each year. Thus, speed of making bibliographic materials 
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■Table 10 



Proportion of Technical Reports Read bay Age 
Method of Identification - May 1984 

(%) 











Method of 


Identification 
















Cited 














Library 




in 










All 




Acces- 




Tech- 








Year 


Read- 




sion 


Col- 


nical 


Printed 


Conputer 




Published 


ings 


Routed 


List 


league 


Reports 


Index 


Search 


Other 


1984 


51% 


63% 


55% 


47% 


30% 


33% 


— 


69% 


1983 


23 


23 


11 


29 


30 


33 


20 


15 


1982 


9 


8 


11 


6 


5 


17 


20 


8 


1980-81 


9 


4 


- 


8 


10 


17 


60 


4 


1975-79 


3 


- 




2 


20 






4 


1970-74 


3 




11 


4 










<1970 


2 


3 


11 


4 


5 








Total 


100 


101 


100 


100 


100 


100 


100 


100 


Average 












1.0 


2.4 


0.7 


Age (Yrs) 


1.4 


0.9 


3.2 


1.8 


2.5 



SCXJRCE: King Research, Inc. Survey of Scientists and Eligineers {n=195) 
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Table 11 

Proportion of Technical Reports Read by 
Method of Idf^ntification by Age - May 1984 

(%) 



Method of Identification 
Cj±ed 

Library in 
All Acces- Tech- 



Year 
Published 


Read- 
ings 


Routed 


SiCHl 

List 


Col- 
league 


nical 
Reports 


Printed 
Index 


Ccnputer 
Search 


Ottier 


Tbtal 


1984 


51% 


26 


6 


30 


7 


3 




29 


101 


1983 


24% 


20 


2 


41 


14 


6 


2 


14 


99 


1982 


8% 


21 


7 


26 


7 


10 


7 


22 


100 


1980-81 


8% 


11 




34 


15 


9 


20 


11 


100 


1975-79 


4% 






37 


60 






3 


100 


1970-74 


2% 




30 


70 










100 


<1970 


3% 


20 


19 


43 


18 








100 


Average 
Age (Yrs) 


1.4 


0.7 


3,2 


1.8 


2.5 


1.0 


2.4 


0.7 





SOURCE: King Research, Inc. Survey of Scientists and Qigineers (n=195} 
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Table 12 



Proportion of Journal Articles Read by Age 
by Method of Identification - May 1984 

(%) 



Method of Identification 



Published 


ings 


Browsing 


Colleague 


Cited in 
AXuicie^ 
etc. 


r'rini.ea 
Index 


"Wilt '1 1 «^ ** 

LAJiii^urei: 
Search 


Other 


1984 


71% 


90% 


42% 


20% 


63% 


19% 


75% 


1983 


12 


7 


24 


13 


24 


25 


10 


1982 


4 


1 


7 


11 


5 


12.5 


6 


1980-81 


4 


0.6 


10 


14 


5 


12.5 


6 


1975-79 


5 


1 


10 


24 


0 


12.5 


3 


1970-74 


2 




3.5 


9 


3 


12.5 




<1970 


2 


0.4 


3.5 


9 


0 


6 


3 


Total 


100 


99 


100 


100 


100 


100 


103 


Average 
Age (Yrs.) 


1.3 


1.0 


2.4 


5.2 


1.2 


4.3 


1.2 



aXJRCE: King Research, Inc. Survey of Scientists and Bigineers (n-664) 
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Table 13 



Proportion of Journal Articles by 
Method of Identification by Age - May 1984 

(%) 



Method of Identification 



Year 
Published 


All 
ings 


Browsing 


Colleague 


Cited in 
Article, 
etc. 


Pr 4nt"p^ 

Index 


fVinriiit"Pr 

Search 


Other 


Tot ad 


1984 


68% 


66 


10 


4 


6 


1 


13 


100 


1983 


12% 


28 


30 


14 


12 


6 


10 


100 


1982 


4% 


11 


25 


33 


7 


9 


14 


99 


1980-81 


5% 


6 


30 


37 


6 


10 


12 


101 


1975-79 


6% 


8 


26 


53 


0 


7 


6 


100 


1970-74 


2% 




23 


52 


8 


17 




100 


<1970 


3% 


8 


22 


47 




8 


15 


100 


Average 
Age (Yrs.) 


1.3 


1.0 


2.4 


5.2 


1.2 


4.3 


1.2 





SOURCE: King Research, Inc. Survey of Scientists and Engineers (n=664} 
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available must be quite good, but sooe improvenjent could possibly be made. 
If one could Increase the publication tijne by, say, two months, the nunber 
of unique readings might ircrease by as much as 24,000 for technical 
reports, and 14,000 for journal articles. Value of these readings, in 
terms of willingness to pay, could be about $3 million for the two together 
emd $22 million in savings. 

2.5 The Effect of Delays in Conducting Bi bliographic Searches on the 
Value of Infonration 

Many bibliograj^ic searches are performed by reference librarians 
{infonnatiai scientists) for DOE (and other) scientists and engineers. 
Another aspect of timeliness that can affect value is the ^jeed with vAiich 
searches are conducted for the ©id users. In Voliane 1, The Value of 
Library Services, we did an in-dQ3th analysis of this aspect of timeliness. 
In that analysis, .*e developed a statistical model (conjoint measurement) 
vrfiich provided a means of analyzing the trade-off of speed of response and 
level of relevance in terms of the value end users would be willing to pay 
measured in their time and, hence, dollars. The results of this trade-off 
analysis are displayec in Figures 1 and 2. 

Figure 1 shows the relative trade-off for manual searches of speed 
of response (between imnediate and four days) and level of relevance of the 
search output. Generally, the reduction in willingness to pay value per 
one day increase in response time is about $6.50. We estijnate that the 
average response time is about 5 days. If this time could be reduced by 
one-half, the value (in willingness to pay) would increase by about $16 per 
eeard Over all manual searches performed for DOE funded scientists and 
engineers, this would come to about $5 million. For computerized 
bibliographic searches (Figure 2) , the reduction in willingness to pay 
value per one day increase in response time is about $8.50. !rhe average 
response time is 3.7 days. If this tijne could be reduced by one-half, the 
value (in willingness to pay) would also increase by about $16 per search 
or about $3 million altogether for all searches. 
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A 

Value to Dscri 
In Terns of Their Ti»e 




Figure Is Value of Manual Searches to Helationahip of Level 
of Relevance and Speed of Response 
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9i3iijxm 2 t telatiocship of Value of Online gcarehes to Level 
Of llelevanee and Speed of Response 



SECTION 3 

VALUE OF aDKPI^HHeiV£3jESS OP THE ENESSGY Dm BASE 



3.1 BacKground 

In this secticn, we examine the cccpr^ensiveness of the Ehergy 
Data Base frcjin the stanc^int of non-D.S. naterials. Of a.11 the readings, 
it is estimated that about 13 percent of the technical report readings and 
15 percent of the journal article readings are from non-U. S. materials. 
!I1ie distribution of these readings over various countries is as follows: 

Table 14 

Proportion of Technical Report and Journal Article Readings 
Tfiat Are Published in Non-U. S. Countries 

{%) 



Gauntry Technical Rqport Readings* Journal Article Readings* 

All Headings Non-U. S. All Readings Non-U. S. 



Canada 


2.3% 


18% 


2.0% 


13% 


Uhited Kingdom 


2.3 


18 


5.1 


34 


France 






0.6 


4 


Germany 


1.6 


12 


2.1 


14 


Japan 


1.6 


12 


1.5 


10 


Russia 


0.8 


6 


0.5 


3 


Other 


4.6 


35 


3.5 


23 


SOURQ:: King 


Research, Inc. 


Survey of Scientists 


and Qigineers 


(n*=: 



n*''=664) 



27 

116 125 



It is clear that the Biglish-speaking countries accx>unt for a large portion 
of the non-U. S. reading, but Germany and Japan also account for about ten 
to 15 percent each. 

It was found that the readers knew about the research prior to 
reading non-U. S. materials in about the same proportions as U.S. materials. 
Fox exairple, in eibout 53 percent of the readings of ncn-U.S. technical 
reports, the reader knew about the research ccmpared with about 59 percait 
for U.S. technical r^rts. For journal article readings, this prc^rtiwi 
of knowledge of research was 44 percent for non-U.S. and 45 percent for 
U.S. articles. However, the information in the materials is found to be 
iaiiqu<^ rrc:e frequently for non-U.S. materials than U.S. materials. For 
example, about three-fourths of the n<xi-U.S. technical reports are found to 
be unique coopared to 63 percent of the U.S. tedinicai reports, and 81 
percent of the non-U.S. journal articles are found to be unique versus 67 
percent for the U.S. articles. This suggests that there might be some 
increase in value for an average reading of non-U.S, technical reports and 
journal articles. 

In terms of savings derived from having read the non-U.S, 
materials, we estimate that about nine percent of the non-U.S. technical 
report readings had savings compared to 48 percent for U.S. technical 
reports. On the other hand, about 20 percent of the ncai-U.S. journal 
article readings yielded savings coipared to 25% for U.S. journal articles. 
Applying these data to readings by DOE-funded scientists and engineers, we 
estimate that there are about 300,000 unique non-U.S. technical report 
readings that yielded savings and 480,000 unique ncn-U.S. journal article 
readings that resulted in savings. ISie next question is how many of these 
readings resulted from searching the EJiergy Data Base, either through the 
Ejieroy Research Abstracts or REOON. 

Applying the proportion of readings attributable to searches from 
the ECB, we estimate that the number of unique non-U.S. technical report 
readings identified from the EIB {£Ba and RECON) that yield savings is 
about 34,000 and the number of unique ncxi-U.S. journal article readings tha 
yield savings is 54,000, The total values in terms of willingness to pay 
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are $3.5 million and $1.9 million for technical r^x>rt and journal article 
readii^Sr re^sectively. Hbe total savings values are $24 million for 
non-U. S. technical rej»rt readings and $21 million for non-U. S. journal 
article readings. When substitutioiib are considered the total value of 
non-U. S. reading comes to $2 million in willingness to pay and $17 million 
in savings value. ITotal non-U. S. material readings from the £3B are 240,000 
technical resort readings and 120,000 journal article readings, or about 
four non-U. S. technical report readings per DOE-funded scientist or 
engineer per year, and two non-U. S. journal article readings per year. 

Of seme interest is how general scientists and engineers identify 
and gain access to non-U. S. materials. Iftis data is surmarized in Table 15 
below. 

Table 15 

Proportion of Readings of Technical Repcrts and Journal Articles 
Identified by Various Methods by U.S. and non-U. S. Materials - 1984 

(%) 



Technical Reports* Journal Articles 
Ideitif icaticn — 



Methods 


U.S. 


Nm-U.S. 


U.S. 


Non-U. S. 


Browsing 






55% 


18% 


Routing 


21% 


21% 






Library Access icns List 


€% 


0% 






Colleague 


34% 


29% 


15% 


18% 


Cited in Other Materials 


10% 


21% 


11% 


26% 


Printed Index 


4% 


7% 


6% 


7% 


Conputerized Search 


2% 


7% 


2% 


10% 


Other 


23% 


14% 


11% 


21% 


Itotal 


100 


100 


6,035,000 


1,065,000 



aDURCE: King Research, Inc. General Survey of Scientists and Engineers 
(n*=195, n**=644) 
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It is evident that non-U.S. technical reports are more likely to be found 
in citations and bibliographic materials than are the U.S. technical 
materials. Oi the other hand, read non-U.S. journal articles are much less 
likely to be found by browsing than read U.S. journal articles. 




SBCTIOU 4 

ocss-cumNG use or JSJFoawnati iKnsm£ 



4.1 Croas-cutting Itee of Materials 

Tte Biergy Data Base covers a wide range of subject areas. A 
question arises about whether such wide-ranging materials should be 
covered. A anall stud^ vas perfonned to provide sane evidence concerning 
this issue. This study involved 21 actual oon|xiterized bibliographic 
searches performed by Department of Ehergy-funded scientists and engineers. 
fl)e scientists or engineers were asked to describe the project for i^ich 
they were searching. They also specified the topics of their search. At a 
later tixne, aibject experts from CSTI were asked to classify the project 
descriptions to provide a cocparison of the subject area of the project and 
the searches that were performed. It turns out that the Hatching of 
project subjects and sezirch subjects is only partially common. !Ihe 
proportion of overlap is shown in the Venn diagram in Figure 3 below. 



subject in 
Kesearch Project Only 




Subject in 
Search Quest i(»i Oily 



Subject Conmon to 
Research Project and Search Quest j.on 



Figure 3 



(Verlap of Subjects Pound in Research Project Title 
and Search Questions (n«21) 



31 

120 



ERIC 



120 



A little more than one-half of the subject classifications were ccnmon to 
both; thus, it would appear that a greater coverage of subject areas is 
needed. 

Another indication is the usefulness of the items retrieved in the 
searches, ihe end-users were asked to indicate (1) v^ich itenis on the 
print-out were of interest to them in terms of the specific research 
project (for which the search was conducted) , (2) the other items that were 
also of general interest to them, and (3) the itenis that they had read or 
intended to read. These itesns were inspected by OHTI and KRI to determine 
how many itesns fell into the classes above. Hie results of the 21 searches 
are given in Table 14. Suanary of results from the table are as follows: 

• lliere was an average of 24 itenis identified per search (hits) 

• Of these, an average of 12.8 were read (or intended to be read) 

• An average of 11.8 of the items identified were thought to be 
relevant to the research project and 7.2 of these were read 

• An average of 12 of the items identified were also of general 
interest to the scientists and engineers and 2.4 of these were 
read. 

ThuSf not only are the subject areas of search different than the research 
project, but they yield a nun4)er of items that are read for the research 
project as well as for general interest. 

Ihe individual items retrieved that were (1) relevant and (2) read 
were also examined to see if they were D.S. or ncxi-U.S. publications. Of 
items that were judged by the users to be relevant to their research 
project and of general interest, it was found that 20 percent of both 
categories were non-U. S. publications. 'Riis conpares to 13 percent for 
technical reports and 15 percent for journal articles observed in the 
general survey of scientists and engineers, 'ttie proportion read is about 
eight percent non-U. S. Again, this is supporting evidence that the 
scientists and engineers funded by DOE tend to use non-U.S. materials more 
frequently than other scientists and engineers. 
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T^le 16 



NUirber of itms Retrieved, Read and Relevant to Project 
and General Interest by D.S. and Non-U. S. Mater iads - 1984 



Seard) 
Hmber 


Total 


Research Project 


Genered. 


Interest 




Read 


U.S. 


Non-U. S. 


U.S. 


Non-U.S. 


U.S. 


Ncm-U.S, 


1 


85 


8 


1 


4 


1 


9 


1 


2 


40 


3 


4 


10 


11 


7 


4 


3 


33 


9 


3 


11 


3 


6 


1 


4 


59 


3 total* 


59 total* 


13 


total* 


5 


60 


6 


6 


23 


7 


14 


3 


6 


72 


17 


25 


1 


3 


3 


2 


7 


47 


3 


1 


3 


1 


11 


1 


8 


61 


11 




11 




15 


- 


9 


35 


10 








10 


- 


10 


63 


27 




18 


3 


2 


- 


11 


128 


1 




5 




5 


- 


12 


31 


9 




7 




10 




13 


29 


4 




12 


7 


9 




14 


20 


8 


1 


2 


1 


4 


1 


15 


15 


7 




9 




12 




16 


28 


4 




1 




7 




17 


11 


7 








7 




18 


30 


3 




7 


1 


1 




19 


92 


39 


7 


5 


1 


37 


3 


20 


249 


17 


3 


9 


1 


32 


5 


21 


124 


1 




15 




35 





* Breakdown by U.S. and non-U. S. not available 

SOURCE: King Research, Inc. -Survey of Online Bibliographic Searches (n=21) 
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